[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250623 01:57:10.864686 12636 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.87.62:36591
I20250623 01:57:10.866156 12636 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:10.867144 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:10.883266 12643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:10.883308 12645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:10.884503 12642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:10.885288 12636 server_base.cc:1048] running on GCE node
I20250623 01:57:12.044474 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:57:12.044703 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:57:12.044893 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643832044876 us; error 0 us; skew 500 ppm
I20250623 01:57:12.045723 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:12.050730 12636 webserver.cc:469] Webserver started at http://127.12.87.62:42127/ using document root <none> and password file <none>
I20250623 01:57:12.051496 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:12.051671 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:12.052035 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:12.057642 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/master-0-root/instance:
uuid: "4fe5f9e622af439fba9683f9c1e27387"
format_stamp: "Formatted at 2025-06-23 01:57:12 on dist-test-slave-47g2"
I20250623 01:57:12.064829 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250623 01:57:12.069625 12651 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.070606 12636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250623 01:57:12.070960 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/master-0-root
uuid: "4fe5f9e622af439fba9683f9c1e27387"
format_stamp: "Formatted at 2025-06-23 01:57:12 on dist-test-slave-47g2"
I20250623 01:57:12.071264 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:12.117937 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:12.119323 12636 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:12.119729 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:12.186712 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.62:36591
I20250623 01:57:12.186789 12702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.62:36591 every 8 connection(s)
I20250623 01:57:12.193375 12703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:12.213776 12703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387: Bootstrap starting.
I20250623 01:57:12.219098 12703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:12.220919 12703 log.cc:826] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:12.225073 12703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387: No bootstrap required, opened a new log
I20250623 01:57:12.243934 12703 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5f9e622af439fba9683f9c1e27387" member_type: VOTER }
I20250623 01:57:12.244414 12703 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:12.244647 12703 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe5f9e622af439fba9683f9c1e27387, State: Initialized, Role: FOLLOWER
I20250623 01:57:12.245500 12703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5f9e622af439fba9683f9c1e27387" member_type: VOTER }
I20250623 01:57:12.245985 12703 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:57:12.246248 12703 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:57:12.246546 12703 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:12.250218 12703 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5f9e622af439fba9683f9c1e27387" member_type: VOTER }
I20250623 01:57:12.250892 12703 leader_election.cc:304] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe5f9e622af439fba9683f9c1e27387; no voters:
I20250623 01:57:12.252405 12703 leader_election.cc:290] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:57:12.252754 12706 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:12.255285 12706 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe5f9e622af439fba9683f9c1e27387, State: Running, Role: LEADER
I20250623 01:57:12.256103 12706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5f9e622af439fba9683f9c1e27387" member_type: VOTER }
I20250623 01:57:12.257135 12703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:57:12.265712 12707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fe5f9e622af439fba9683f9c1e27387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5f9e622af439fba9683f9c1e27387" member_type: VOTER } }
I20250623 01:57:12.265856 12708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe5f9e622af439fba9683f9c1e27387. Latest consensus state: current_term: 1 leader_uuid: "4fe5f9e622af439fba9683f9c1e27387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe5f9e622af439fba9683f9c1e27387" member_type: VOTER } }
I20250623 01:57:12.266414 12707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:12.266494 12708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:12.269778 12715 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:57:12.279510 12715 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:57:12.281643 12636 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250623 01:57:12.293987 12715 catalog_manager.cc:1349] Generated new cluster ID: 6862d62ba4af467cbcd94638527fdac7
I20250623 01:57:12.294261 12715 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:57:12.309356 12715 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:57:12.311280 12715 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:57:12.336069 12715 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387: Generated new TSK 0
I20250623 01:57:12.337111 12715 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:57:12.347862 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:12.355521 12725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:12.356482 12726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:12.360255 12728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:12.361013 12636 server_base.cc:1048] running on GCE node
I20250623 01:57:12.361899 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:57:12.362149 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:57:12.362349 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643832362331 us; error 0 us; skew 500 ppm
I20250623 01:57:12.363134 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:12.367041 12636 webserver.cc:469] Webserver started at http://127.12.87.1:39981/ using document root <none> and password file <none>
I20250623 01:57:12.367693 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:12.367930 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:12.368234 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:12.369617 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-0-root/instance:
uuid: "1941a8d154b24a47be40c5f39bba84df"
format_stamp: "Formatted at 2025-06-23 01:57:12 on dist-test-slave-47g2"
I20250623 01:57:12.376197 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250623 01:57:12.381040 12733 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.381865 12636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250623 01:57:12.382184 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-0-root
uuid: "1941a8d154b24a47be40c5f39bba84df"
format_stamp: "Formatted at 2025-06-23 01:57:12 on dist-test-slave-47g2"
I20250623 01:57:12.382534 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:12.408501 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:12.410441 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:12.413200 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:12.418491 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:12.418809 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.419107 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:12.419329 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.504477 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.1:34427
I20250623 01:57:12.504482 12795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.1:34427 every 8 connection(s)
I20250623 01:57:12.512081 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:12.522461 12800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:12.525844 12801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:12.538844 12796 heartbeater.cc:344] Connected to a master server at 127.12.87.62:36591
I20250623 01:57:12.539352 12796 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:12.540230 12636 server_base.cc:1048] running on GCE node
W20250623 01:57:12.540287 12803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:12.540541 12796 heartbeater.cc:507] Master 127.12.87.62:36591 requested a full tablet report, sending...
I20250623 01:57:12.541473 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:57:12.541738 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:57:12.541949 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643832541931 us; error 0 us; skew 500 ppm
I20250623 01:57:12.542579 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:12.543746 12668 ts_manager.cc:194] Registered new tserver with Master: 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427)
I20250623 01:57:12.546196 12636 webserver.cc:469] Webserver started at http://127.12.87.2:37343/ using document root <none> and password file <none>
I20250623 01:57:12.546686 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:12.546875 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:12.546813 12668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51634
I20250623 01:57:12.547222 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:12.548482 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-1-root/instance:
uuid: "08809c450cf0440dac8d4a2348b94d3c"
format_stamp: "Formatted at 2025-06-23 01:57:12 on dist-test-slave-47g2"
I20250623 01:57:12.553658 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20250623 01:57:12.557765 12808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.558540 12636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250623 01:57:12.558787 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-1-root
uuid: "08809c450cf0440dac8d4a2348b94d3c"
format_stamp: "Formatted at 2025-06-23 01:57:12 on dist-test-slave-47g2"
I20250623 01:57:12.559042 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:12.575769 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:12.576968 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:12.578374 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:12.581502 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:12.581671 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.581888 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:12.582018 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.686146 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.2:39703
I20250623 01:57:12.686492 12870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.2:39703 every 8 connection(s)
I20250623 01:57:12.692600 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:12.700853 12874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:12.706903 12871 heartbeater.cc:344] Connected to a master server at 127.12.87.62:36591
I20250623 01:57:12.707408 12871 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:12.708593 12871 heartbeater.cc:507] Master 127.12.87.62:36591 requested a full tablet report, sending...
I20250623 01:57:12.711308 12668 ts_manager.cc:194] Registered new tserver with Master: 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:12.713442 12668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51650
W20250623 01:57:12.718338 12875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:12.719808 12877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:12.719983 12636 server_base.cc:1048] running on GCE node
I20250623 01:57:12.720883 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:57:12.721077 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:57:12.721243 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643832721225 us; error 0 us; skew 500 ppm
I20250623 01:57:12.721784 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:12.724575 12636 webserver.cc:469] Webserver started at http://127.12.87.3:42759/ using document root <none> and password file <none>
I20250623 01:57:12.725035 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:12.725230 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:12.725502 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:12.726464 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-2-root/instance:
uuid: "8974a68eaebc4629b7ff34eeb07183bf"
format_stamp: "Formatted at 2025-06-23 01:57:12 on dist-test-slave-47g2"
I20250623 01:57:12.730688 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20250623 01:57:12.733690 12882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.734422 12636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250623 01:57:12.734660 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-2-root
uuid: "8974a68eaebc4629b7ff34eeb07183bf"
format_stamp: "Formatted at 2025-06-23 01:57:12 on dist-test-slave-47g2"
I20250623 01:57:12.734958 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750643830703842-12636-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:12.756892 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:12.758227 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:12.760809 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:12.764065 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:12.764245 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.764498 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:12.764638 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:12.856880 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.3:34451
I20250623 01:57:12.856976 12944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.3:34451 every 8 connection(s)
I20250623 01:57:12.875809 12945 heartbeater.cc:344] Connected to a master server at 127.12.87.62:36591
I20250623 01:57:12.876173 12945 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:12.876919 12945 heartbeater.cc:507] Master 127.12.87.62:36591 requested a full tablet report, sending...
I20250623 01:57:12.878655 12668 ts_manager.cc:194] Registered new tserver with Master: 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:12.879217 12636 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013280012s
I20250623 01:57:12.880344 12668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51664
I20250623 01:57:12.883165 12636 test_util.cc:276] Using random seed: -2016835642
I20250623 01:57:12.926573 12668 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51674:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 01:57:12.929980 12668 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 01:57:13.026803 12836 tablet_service.cc:1468] Processing CreateTablet for tablet 8034e2a5ed434721b58a76f8d8310e33 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250623 01:57:13.027185 12760 tablet_service.cc:1468] Processing CreateTablet for tablet 7177b120b41a44e386b315897a02e2b3 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250623 01:57:13.028705 12760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7177b120b41a44e386b315897a02e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.029021 12835 tablet_service.cc:1468] Processing CreateTablet for tablet 7177b120b41a44e386b315897a02e2b3 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250623 01:57:13.030535 12759 tablet_service.cc:1468] Processing CreateTablet for tablet c52d16a94d354ba290e8e7e4335a548b (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250623 01:57:13.030946 12834 tablet_service.cc:1468] Processing CreateTablet for tablet c52d16a94d354ba290e8e7e4335a548b (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250623 01:57:13.031927 12759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52d16a94d354ba290e8e7e4335a548b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.032754 12833 tablet_service.cc:1468] Processing CreateTablet for tablet 5442a9db195b424a8cbb938180e80a45 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250623 01:57:13.038646 12836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8034e2a5ed434721b58a76f8d8310e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.037703 12761 tablet_service.cc:1468] Processing CreateTablet for tablet 8034e2a5ed434721b58a76f8d8310e33 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250623 01:57:13.047137 12830 tablet_service.cc:1468] Processing CreateTablet for tablet b6c40a8fa54e45b195056cb49339ac8f (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250623 01:57:13.050209 12756 tablet_service.cc:1468] Processing CreateTablet for tablet 59a83d61adce49d8ab74a9f4ddab4cf3 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250623 01:57:13.042646 12832 tablet_service.cc:1468] Processing CreateTablet for tablet c79edf8d51584557b617641cdb0c5908 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250623 01:57:13.050155 12758 tablet_service.cc:1468] Processing CreateTablet for tablet 5442a9db195b424a8cbb938180e80a45 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250623 01:57:13.039319 12753 tablet_service.cc:1468] Processing CreateTablet for tablet 6093f636863d4e2fb12280c21a63be5a (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250623 01:57:13.059567 12757 tablet_service.cc:1468] Processing CreateTablet for tablet c79edf8d51584557b617641cdb0c5908 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250623 01:57:13.053925 12835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7177b120b41a44e386b315897a02e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.045565 12754 tablet_service.cc:1468] Processing CreateTablet for tablet e737c71979a149d98198bcd4308ff967 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250623 01:57:13.053350 12755 tablet_service.cc:1468] Processing CreateTablet for tablet b6c40a8fa54e45b195056cb49339ac8f (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250623 01:57:13.045603 12831 tablet_service.cc:1468] Processing CreateTablet for tablet 59a83d61adce49d8ab74a9f4ddab4cf3 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250623 01:57:13.075166 12761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8034e2a5ed434721b58a76f8d8310e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.080176 12758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5442a9db195b424a8cbb938180e80a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.081930 12832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79edf8d51584557b617641cdb0c5908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.083415 12833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5442a9db195b424a8cbb938180e80a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.085366 12753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6093f636863d4e2fb12280c21a63be5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.086932 12834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52d16a94d354ba290e8e7e4335a548b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.088348 12757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79edf8d51584557b617641cdb0c5908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.089838 12830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c40a8fa54e45b195056cb49339ac8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.091322 12754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e737c71979a149d98198bcd4308ff967. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.092782 12756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a83d61adce49d8ab74a9f4ddab4cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.094234 12755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c40a8fa54e45b195056cb49339ac8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.095722 12831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a83d61adce49d8ab74a9f4ddab4cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.106619 12829 tablet_service.cc:1468] Processing CreateTablet for tablet e737c71979a149d98198bcd4308ff967 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250623 01:57:13.107978 12829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e737c71979a149d98198bcd4308ff967. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.125881 12963 tablet_bootstrap.cc:492] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.130789 12964 tablet_bootstrap.cc:492] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.139452 12963 tablet_bootstrap.cc:654] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.148147 12829 tablet_service.cc:1468] Processing CreateTablet for tablet 6093f636863d4e2fb12280c21a63be5a (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250623 01:57:13.150624 12829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6093f636863d4e2fb12280c21a63be5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.191840 12909 tablet_service.cc:1468] Processing CreateTablet for tablet 7177b120b41a44e386b315897a02e2b3 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250623 01:57:13.200115 12907 tablet_service.cc:1468] Processing CreateTablet for tablet 5442a9db195b424a8cbb938180e80a45 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250623 01:57:13.196590 12910 tablet_service.cc:1468] Processing CreateTablet for tablet 8034e2a5ed434721b58a76f8d8310e33 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250623 01:57:13.204380 12905 tablet_service.cc:1468] Processing CreateTablet for tablet 59a83d61adce49d8ab74a9f4ddab4cf3 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250623 01:57:13.198655 12908 tablet_service.cc:1468] Processing CreateTablet for tablet c52d16a94d354ba290e8e7e4335a548b (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250623 01:57:13.199997 12906 tablet_service.cc:1468] Processing CreateTablet for tablet c79edf8d51584557b617641cdb0c5908 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250623 01:57:13.211361 12907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5442a9db195b424a8cbb938180e80a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.213127 12905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a83d61adce49d8ab74a9f4ddab4cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.217162 12902 tablet_service.cc:1468] Processing CreateTablet for tablet 6093f636863d4e2fb12280c21a63be5a (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250623 01:57:13.218056 12903 tablet_service.cc:1468] Processing CreateTablet for tablet e737c71979a149d98198bcd4308ff967 (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250623 01:57:13.232003 12910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8034e2a5ed434721b58a76f8d8310e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.217993 12904 tablet_service.cc:1468] Processing CreateTablet for tablet b6c40a8fa54e45b195056cb49339ac8f (DEFAULT_TABLE table=test-workload [id=731c246e952142d5a36ad9d960c01a58]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250623 01:57:13.239507 12908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52d16a94d354ba290e8e7e4335a548b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.242630 12903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e737c71979a149d98198bcd4308ff967. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.243770 12904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c40a8fa54e45b195056cb49339ac8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.245891 12909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7177b120b41a44e386b315897a02e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.252310 12906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79edf8d51584557b617641cdb0c5908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:13.273487 12902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6093f636863d4e2fb12280c21a63be5a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 01:57:13.276492 12716 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:57:13.286840 12964 tablet_bootstrap.cc:654] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.284940 12963 tablet_bootstrap.cc:492] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.292905 12963 ts_tablet_manager.cc:1397] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.168s user 0.009s sys 0.016s
I20250623 01:57:13.320673 12966 tablet_bootstrap.cc:492] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.324683 12963 raft_consensus.cc:357] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.327092 12963 raft_consensus.cc:383] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.327457 12963 raft_consensus.cc:738] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.328450 12963 consensus_queue.cc:260] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.348439 12796 heartbeater.cc:499] Master 127.12.87.62:36591 was elected leader, sending a full tablet report...
I20250623 01:57:13.357018 12963 ts_tablet_manager.cc:1428] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.045s user 0.007s sys 0.005s
I20250623 01:57:13.362143 12963 tablet_bootstrap.cc:492] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.359596 12966 tablet_bootstrap.cc:654] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.366978 12963 tablet_bootstrap.cc:654] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.360086 12964 tablet_bootstrap.cc:492] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.369490 12964 ts_tablet_manager.cc:1397] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.239s user 0.014s sys 0.006s
I20250623 01:57:13.372941 12964 raft_consensus.cc:357] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.374049 12964 raft_consensus.cc:383] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.374439 12964 raft_consensus.cc:738] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.375298 12964 consensus_queue.cc:260] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.381299 12966 tablet_bootstrap.cc:492] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.381826 12966 ts_tablet_manager.cc:1397] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.002s
I20250623 01:57:13.381984 12871 heartbeater.cc:499] Master 127.12.87.62:36591 was elected leader, sending a full tablet report...
I20250623 01:57:13.382673 12964 ts_tablet_manager.cc:1428] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.013s user 0.006s sys 0.003s
I20250623 01:57:13.383581 12964 tablet_bootstrap.cc:492] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.384560 12966 raft_consensus.cc:357] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.385396 12966 raft_consensus.cc:383] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.385733 12966 raft_consensus.cc:738] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.386579 12966 consensus_queue.cc:260] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.387987 12963 tablet_bootstrap.cc:492] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.388415 12963 ts_tablet_manager.cc:1397] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.000s
I20250623 01:57:13.390842 12964 tablet_bootstrap.cc:654] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.391013 12963 raft_consensus.cc:357] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.391744 12963 raft_consensus.cc:383] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.392056 12963 raft_consensus.cc:738] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.392760 12963 consensus_queue.cc:260] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.393808 12967 raft_consensus.cc:491] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.394405 12967 raft_consensus.cc:513] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.400664 12967 leader_election.cc:290] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.437193 12945 heartbeater.cc:499] Master 127.12.87.62:36591 was elected leader, sending a full tablet report...
I20250623 01:57:13.437877 12966 ts_tablet_manager.cc:1428] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.056s user 0.005s sys 0.026s
I20250623 01:57:13.438493 12964 tablet_bootstrap.cc:492] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.438751 12966 tablet_bootstrap.cc:492] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.438906 12964 ts_tablet_manager.cc:1397] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.011s
W20250623 01:57:13.444479 12872 tablet.cc:2378] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:57:13.446404 12963 ts_tablet_manager.cc:1428] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.058s user 0.007s sys 0.009s
I20250623 01:57:13.447443 12963 tablet_bootstrap.cc:492] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.447952 12966 tablet_bootstrap.cc:654] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.454275 12964 raft_consensus.cc:357] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.455549 12964 raft_consensus.cc:383] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.456156 12964 raft_consensus.cc:738] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.456921 12964 consensus_queue.cc:260] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.458470 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:13.460676 12964 ts_tablet_manager.cc:1428] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.022s user 0.005s sys 0.003s
W20250623 01:57:13.462085 12737 leader_election.cc:343] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.463799 12964 tablet_bootstrap.cc:492] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.465090 12966 tablet_bootstrap.cc:492] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.465591 12966 ts_tablet_manager.cc:1397] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.004s
I20250623 01:57:13.467386 12963 tablet_bootstrap.cc:654] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.472558 12966 raft_consensus.cc:357] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.473866 12964 tablet_bootstrap.cc:654] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.473927 12966 raft_consensus.cc:383] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.472846 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:13.474287 12966 raft_consensus.cc:738] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.475000 12966 consensus_queue.cc:260] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
W20250623 01:57:13.475653 12734 leader_election.cc:343] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.476110 12734 leader_election.cc:304] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df; no voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:13.476991 12973 raft_consensus.cc:2747] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:13.478235 12966 ts_tablet_manager.cc:1428] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20250623 01:57:13.479221 12966 tablet_bootstrap.cc:492] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.488668 12963 tablet_bootstrap.cc:492] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.489286 12963 ts_tablet_manager.cc:1397] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.007s
I20250623 01:57:13.493289 12963 raft_consensus.cc:357] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.493973 12963 raft_consensus.cc:383] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.494208 12963 raft_consensus.cc:738] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.494791 12966 tablet_bootstrap.cc:654] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.495005 12963 consensus_queue.cc:260] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.495719 12964 tablet_bootstrap.cc:492] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.496204 12964 ts_tablet_manager.cc:1397] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.009s
I20250623 01:57:13.498817 12964 raft_consensus.cc:357] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.499531 12964 raft_consensus.cc:383] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.499850 12964 raft_consensus.cc:738] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.500563 12964 consensus_queue.cc:260] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.502007 12971 raft_consensus.cc:491] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.502465 12971 raft_consensus.cc:513] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.504933 12971 leader_election.cc:290] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.507133 12964 ts_tablet_manager.cc:1428] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250623 01:57:13.508114 12964 tablet_bootstrap.cc:492] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.514879 12964 tablet_bootstrap.cc:654] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.516330 12963 ts_tablet_manager.cc:1428] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.027s user 0.003s sys 0.003s
I20250623 01:57:13.522802 12966 tablet_bootstrap.cc:492] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.525013 12966 ts_tablet_manager.cc:1397] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.046s user 0.016s sys 0.016s
I20250623 01:57:13.525468 12963 tablet_bootstrap.cc:492] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.527621 12966 raft_consensus.cc:357] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.528301 12966 raft_consensus.cc:383] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.528558 12966 raft_consensus.cc:738] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.529166 12966 consensus_queue.cc:260] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.531606 12963 tablet_bootstrap.cc:654] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.535187 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df" is_pre_election: true
W20250623 01:57:13.536674 12809 leader_election.cc:343] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.538826 12966 ts_tablet_manager.cc:1428] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.014s user 0.003s sys 0.003s
I20250623 01:57:13.539793 12966 tablet_bootstrap.cc:492] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.542407 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:13.545868 12966 tablet_bootstrap.cc:654] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
W20250623 01:57:13.549619 12809 leader_election.cc:343] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.550274 12809 leader_election.cc:304] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c; no voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:13.552117 12964 tablet_bootstrap.cc:492] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.552587 12981 raft_consensus.cc:2747] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:13.554749 12964 ts_tablet_manager.cc:1397] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.000s
I20250623 01:57:13.556772 12966 tablet_bootstrap.cc:492] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.557116 12966 ts_tablet_manager.cc:1397] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20250623 01:57:13.554296 12963 tablet_bootstrap.cc:492] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.557683 12963 ts_tablet_manager.cc:1397] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.003s
I20250623 01:57:13.557433 12964 raft_consensus.cc:357] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.558207 12964 raft_consensus.cc:383] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.558557 12964 raft_consensus.cc:738] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.559281 12964 consensus_queue.cc:260] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.559753 12966 raft_consensus.cc:357] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.560509 12966 raft_consensus.cc:383] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.560873 12966 raft_consensus.cc:738] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.561657 12964 ts_tablet_manager.cc:1428] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250623 01:57:13.561643 12966 consensus_queue.cc:260] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.562722 12964 tablet_bootstrap.cc:492] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.564231 12966 ts_tablet_manager.cc:1428] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250623 01:57:13.565198 12966 tablet_bootstrap.cc:492] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.569327 12964 tablet_bootstrap.cc:654] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.571604 12966 tablet_bootstrap.cc:654] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.580749 12966 tablet_bootstrap.cc:492] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.581219 12966 ts_tablet_manager.cc:1397] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20250623 01:57:13.579480 12963 raft_consensus.cc:357] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.582811 12963 raft_consensus.cc:383] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.583122 12963 raft_consensus.cc:738] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.583873 12966 raft_consensus.cc:357] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.584558 12966 raft_consensus.cc:383] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.584208 12963 consensus_queue.cc:260] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.584827 12966 raft_consensus.cc:738] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.586311 12966 consensus_queue.cc:260] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.586895 12964 tablet_bootstrap.cc:492] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.587920 12972 raft_consensus.cc:491] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.588359 12972 raft_consensus.cc:513] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.590708 12972 leader_election.cc:290] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427)
I20250623 01:57:13.592340 12966 ts_tablet_manager.cc:1428] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250623 01:57:13.593293 12966 tablet_bootstrap.cc:492] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.586768 12963 ts_tablet_manager.cc:1428] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250623 01:57:13.599406 12966 tablet_bootstrap.cc:654] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.609122 12963 tablet_bootstrap.cc:492] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.610999 12966 tablet_bootstrap.cc:492] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.611174 12964 ts_tablet_manager.cc:1397] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.007s
I20250623 01:57:13.611465 12966 ts_tablet_manager.cc:1397] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.013s
I20250623 01:57:13.615509 12964 raft_consensus.cc:357] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.618562 12964 raft_consensus.cc:383] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.617956 12963 tablet_bootstrap.cc:654] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.618913 12964 raft_consensus.cc:738] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.619639 12964 consensus_queue.cc:260] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.622022 12964 ts_tablet_manager.cc:1428] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250623 01:57:13.622917 12964 tablet_bootstrap.cc:492] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.617945 12966 raft_consensus.cc:357] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.629451 12966 raft_consensus.cc:383] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.629797 12966 raft_consensus.cc:738] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.630450 12966 consensus_queue.cc:260] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.632668 12966 ts_tablet_manager.cc:1428] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20250623 01:57:13.633540 12966 tablet_bootstrap.cc:492] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.636817 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df" is_pre_election: true
I20250623 01:57:13.637549 12771 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 0.
I20250623 01:57:13.637693 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:13.638247 12846 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 0.
I20250623 01:57:13.640880 12963 tablet_bootstrap.cc:492] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.641373 12963 ts_tablet_manager.cc:1397] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.006s
I20250623 01:57:13.641705 12966 tablet_bootstrap.cc:654] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.643882 12883 leader_election.cc:304] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:13.644233 12963 raft_consensus.cc:357] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.645188 12972 raft_consensus.cc:2802] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:13.645260 12963 raft_consensus.cc:383] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.645617 12972 raft_consensus.cc:491] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:13.645711 12963 raft_consensus.cc:738] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.646001 12972 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.646135 12964 tablet_bootstrap.cc:654] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.646502 12963 consensus_queue.cc:260] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.648880 12963 ts_tablet_manager.cc:1428] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250623 01:57:13.650015 12963 tablet_bootstrap.cc:492] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.652233 12972 raft_consensus.cc:513] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.654115 12972 leader_election.cc:290] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427)
I20250623 01:57:13.659790 12963 tablet_bootstrap.cc:654] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.660102 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:13.660395 12966 tablet_bootstrap.cc:492] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.660661 12846 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.660877 12966 ts_tablet_manager.cc:1397] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.006s
I20250623 01:57:13.663841 12966 raft_consensus.cc:357] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.664688 12966 raft_consensus.cc:383] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.665048 12966 raft_consensus.cc:738] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.665910 12966 consensus_queue.cc:260] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.669276 12964 tablet_bootstrap.cc:492] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.670214 12964 ts_tablet_manager.cc:1397] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.014s
I20250623 01:57:13.672433 12963 tablet_bootstrap.cc:492] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.672900 12963 ts_tablet_manager.cc:1397] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.004s
I20250623 01:57:13.672849 12964 raft_consensus.cc:357] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.673650 12964 raft_consensus.cc:383] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.673996 12964 raft_consensus.cc:738] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.674098 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:13.674638 12771 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.674787 12964 consensus_queue.cc:260] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.675565 12963 raft_consensus.cc:357] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.676239 12963 raft_consensus.cc:383] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.676512 12963 raft_consensus.cc:738] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.677134 12963 consensus_queue.cc:260] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.678287 12966 ts_tablet_manager.cc:1428] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.017s user 0.005s sys 0.004s
I20250623 01:57:13.679749 12963 ts_tablet_manager.cc:1428] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250623 01:57:13.678727 12964 ts_tablet_manager.cc:1428] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250623 01:57:13.680017 12966 tablet_bootstrap.cc:492] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.680289 12973 raft_consensus.cc:491] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.680877 12973 raft_consensus.cc:513] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.682863 12973 leader_election.cc:290] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.683837 12963 tablet_bootstrap.cc:492] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.684981 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6093f636863d4e2fb12280c21a63be5a" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:13.685664 12845 raft_consensus.cc:2466] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.689949 12771 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 1.
I20250623 01:57:13.690582 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6093f636863d4e2fb12280c21a63be5a" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:13.690771 12964 tablet_bootstrap.cc:492] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.691187 12920 raft_consensus.cc:2466] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.692462 12737 leader_election.cc:304] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:13.691134 12883 leader_election.cc:304] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:13.694725 12973 raft_consensus.cc:491] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.695318 12973 raft_consensus.cc:513] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.691854 12846 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 1.
I20250623 01:57:13.696409 12972 raft_consensus.cc:2802] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:13.697358 12973 leader_election.cc:290] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.697958 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
W20250623 01:57:13.720456 12737 leader_election.cc:343] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.720664 12963 tablet_bootstrap.cc:654] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.697813 12967 raft_consensus.cc:2802] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:13.721519 12967 raft_consensus.cc:491] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:13.722004 12967 raft_consensus.cc:3058] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.722975 12981 raft_consensus.cc:491] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.723526 12981 raft_consensus.cc:513] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.724463 12972 raft_consensus.cc:695] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:13.725354 12972 consensus_queue.cc:237] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.725950 12981 leader_election.cc:290] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.727662 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df" is_pre_election: true
I20250623 01:57:13.727574 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:13.728526 12920 raft_consensus.cc:2466] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.729585 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:13.729840 12734 leader_election.cc:304] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters: 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:13.730909 12973 raft_consensus.cc:2802] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250623 01:57:13.731051 12809 leader_election.cc:343] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.731272 12973 raft_consensus.cc:491] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:13.731607 12973 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:57:13.731678 12809 leader_election.cc:343] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.732256 12964 tablet_bootstrap.cc:654] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.732074 12809 leader_election.cc:304] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c; no voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:13.733575 12981 raft_consensus.cc:2747] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:13.738118 12967 raft_consensus.cc:513] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.738644 12973 raft_consensus.cc:513] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.739306 12966 tablet_bootstrap.cc:654] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.740728 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6093f636863d4e2fb12280c21a63be5a" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:13.741273 12845 raft_consensus.cc:3058] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.741503 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:13.742107 12919 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.742571 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
W20250623 01:57:13.743786 12737 leader_election.cc:343] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.749198 12919 raft_consensus.cc:2466] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 1.
I20250623 01:57:13.749974 12964 tablet_bootstrap.cc:492] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.750450 12964 ts_tablet_manager.cc:1397] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.004s
I20250623 01:57:13.750362 12734 leader_election.cc:304] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters: 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:13.750941 12845 raft_consensus.cc:2466] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 1.
I20250623 01:57:13.752079 12737 leader_election.cc:304] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:13.753273 12964 raft_consensus.cc:357] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.754022 12964 raft_consensus.cc:383] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.754326 12964 raft_consensus.cc:738] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.754422 12668 catalog_manager.cc:5582] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 1 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:13.757073 12967 leader_election.cc:290] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.758517 12967 raft_consensus.cc:2802] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:13.757302 12963 tablet_bootstrap.cc:492] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.759799 12967 raft_consensus.cc:695] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:13.755065 12964 consensus_queue.cc:260] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.760111 12963 ts_tablet_manager.cc:1397] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.007s
I20250623 01:57:13.757788 12973 leader_election.cc:290] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.761641 12973 raft_consensus.cc:2802] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:13.762094 12973 raft_consensus.cc:695] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:13.762885 12973 consensus_queue.cc:237] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.772447 12963 raft_consensus.cc:357] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.764746 12967 consensus_queue.cc:237] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.766013 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6093f636863d4e2fb12280c21a63be5a" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:13.775192 12919 raft_consensus.cc:3058] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.764008 12964 ts_tablet_manager.cc:1428] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250623 01:57:13.782337 12964 tablet_bootstrap.cc:492] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.774852 12963 raft_consensus.cc:383] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.785768 12963 raft_consensus.cc:738] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.786959 12972 raft_consensus.cc:491] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.787410 12972 raft_consensus.cc:513] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.789256 12972 leader_election.cc:290] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:13.790282 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:13.791023 12845 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 0.
I20250623 01:57:13.791594 12964 tablet_bootstrap.cc:654] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.792261 12886 leader_election.cc:304] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:13.793219 12919 raft_consensus.cc:2466] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 1.
I20250623 01:57:13.831780 12966 tablet_bootstrap.cc:492] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.793467 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df" is_pre_election: true
I20250623 01:57:13.832854 12966 ts_tablet_manager.cc:1397] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.153s user 0.005s sys 0.015s
I20250623 01:57:13.833040 12771 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 0.
I20250623 01:57:13.805630 12972 raft_consensus.cc:491] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.834210 12972 raft_consensus.cc:513] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.834997 12996 raft_consensus.cc:2802] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:13.835379 12996 raft_consensus.cc:491] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:13.835749 12996 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.836301 12972 leader_election.cc:290] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:13.830497 12963 consensus_queue.cc:260] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.838954 12963 ts_tablet_manager.cc:1428] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.078s user 0.016s sys 0.003s
I20250623 01:57:13.839910 12963 tablet_bootstrap.cc:492] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.842370 12966 raft_consensus.cc:357] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.842587 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:13.843016 12966 raft_consensus.cc:383] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.843480 12845 raft_consensus.cc:2466] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 0.
I20250623 01:57:13.843498 12966 raft_consensus.cc:738] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.845088 12886 leader_election.cc:304] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:13.844774 12966 consensus_queue.cc:260] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.846102 12972 raft_consensus.cc:2802] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:13.846434 12972 raft_consensus.cc:491] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:13.846462 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df" is_pre_election: true
I20250623 01:57:13.846781 12972 raft_consensus.cc:3058] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:57:13.847894 12883 leader_election.cc:343] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.852918 12972 raft_consensus.cc:513] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.854948 12972 leader_election.cc:290] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:13.855731 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
W20250623 01:57:13.857232 12883 leader_election.cc:343] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:13.855814 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:13.858060 12845 raft_consensus.cc:3058] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.858393 12963 tablet_bootstrap.cc:654] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.843858 12996 raft_consensus.cc:513] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.861768 12964 tablet_bootstrap.cc:492] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.862701 12966 ts_tablet_manager.cc:1428] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20250623 01:57:13.863430 12964 ts_tablet_manager.cc:1397] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.081s user 0.019s sys 0.018s
I20250623 01:57:13.863988 12966 tablet_bootstrap.cc:492] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf: Bootstrap starting.
I20250623 01:57:13.865113 12845 raft_consensus.cc:2466] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 1.
I20250623 01:57:13.865865 12964 raft_consensus.cc:357] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.866206 12886 leader_election.cc:304] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters: 1941a8d154b24a47be40c5f39bba84df
I20250623 01:57:13.866501 12964 raft_consensus.cc:383] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.866735 12964 raft_consensus.cc:738] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.867089 13008 raft_consensus.cc:2802] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:13.867635 13008 raft_consensus.cc:695] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:13.869341 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:13.869905 12966 tablet_bootstrap.cc:654] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.869998 12771 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.870296 13008 consensus_queue.cc:237] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.867651 12964 consensus_queue.cc:260] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.878916 12964 ts_tablet_manager.cc:1428] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250623 01:57:13.879691 12964 tablet_bootstrap.cc:492] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c: Bootstrap starting.
I20250623 01:57:13.880661 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:13.881518 12845 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.879808 12996 leader_election.cc:290] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:13.885180 12771 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 1.
I20250623 01:57:13.886547 12883 leader_election.cc:304] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:13.887387 12996 raft_consensus.cc:2802] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:13.888010 12996 raft_consensus.cc:695] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:13.888556 12963 tablet_bootstrap.cc:492] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.889014 12963 ts_tablet_manager.cc:1397] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.006s
I20250623 01:57:13.888816 12996 consensus_queue.cc:237] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.891588 12963 raft_consensus.cc:357] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.891322 12667 catalog_manager.cc:5582] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 0 to 1, leader changed from <none> to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 1 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:13.895030 12845 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 1.
I20250623 01:57:13.899148 12964 tablet_bootstrap.cc:654] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.892290 12963 raft_consensus.cc:383] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.899516 12963 raft_consensus.cc:738] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.900278 12963 consensus_queue.cc:260] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.903230 12963 ts_tablet_manager.cc:1428] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.014s user 0.002s sys 0.003s
I20250623 01:57:13.904309 12963 tablet_bootstrap.cc:492] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df: Bootstrap starting.
I20250623 01:57:13.910346 12667 catalog_manager.cc:5582] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 0 to 1, leader changed from <none> to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 1 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:13.911291 12963 tablet_bootstrap.cc:654] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:13.921301 12665 catalog_manager.cc:5582] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 1 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:13.923691 12665 catalog_manager.cc:5582] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 1 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:13.946419 12967 raft_consensus.cc:491] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.946913 12967 raft_consensus.cc:513] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.948727 12967 leader_election.cc:290] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.950489 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:13.951041 12919 raft_consensus.cc:2466] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.951944 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:13.952457 12845 raft_consensus.cc:2466] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.953533 12737 leader_election.cc:304] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:13.954998 12967 raft_consensus.cc:2802] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:13.955387 12967 raft_consensus.cc:491] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:13.955732 12967 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.959167 12973 raft_consensus.cc:491] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.959262 12966 tablet_bootstrap.cc:492] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf: No bootstrap required, opened a new log
I20250623 01:57:13.959852 12966 ts_tablet_manager.cc:1397] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.005s
I20250623 01:57:13.959612 12973 raft_consensus.cc:513] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.961576 12973 leader_election.cc:290] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.962922 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:13.963693 12973 raft_consensus.cc:491] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:13.963449 12845 raft_consensus.cc:2466] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.963267 12966 raft_consensus.cc:357] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.964680 12973 raft_consensus.cc:513] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.965152 12966 raft_consensus.cc:383] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.965488 12737 leader_election.cc:304] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:13.966565 12973 leader_election.cc:290] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.966778 13002 raft_consensus.cc:2802] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:13.965555 12966 raft_consensus.cc:738] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.967137 13002 raft_consensus.cc:491] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:13.967464 13002 raft_consensus.cc:3058] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.967587 12966 consensus_queue.cc:260] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.970134 12964 tablet_bootstrap.cc:492] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c: No bootstrap required, opened a new log
I20250623 01:57:13.970582 12964 ts_tablet_manager.cc:1397] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c: Time spent bootstrapping tablet: real 0.091s user 0.032s sys 0.031s
I20250623 01:57:13.970417 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:13.971549 12920 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.972671 12734 leader_election.cc:304] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:13.973424 12973 raft_consensus.cc:2802] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:13.973749 12973 raft_consensus.cc:491] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:13.974048 12973 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.976012 12966 ts_tablet_manager.cc:1428] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250623 01:57:13.976828 12964 raft_consensus.cc:357] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.977559 12964 raft_consensus.cc:383] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.977713 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:13.963937 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:13.979086 12919 raft_consensus.cc:2466] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.977875 12964 raft_consensus.cc:738] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.980821 12973 raft_consensus.cc:513] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.981338 12964 consensus_queue.cc:260] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.982256 12845 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 0.
I20250623 01:57:13.981738 13002 raft_consensus.cc:513] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.983477 12964 ts_tablet_manager.cc:1428] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250623 01:57:13.985719 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:13.986382 12846 raft_consensus.cc:3058] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.987370 12963 tablet_bootstrap.cc:492] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df: No bootstrap required, opened a new log
I20250623 01:57:13.987795 12963 ts_tablet_manager.cc:1397] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.024s
I20250623 01:57:13.988497 12973 leader_election.cc:290] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:13.989156 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:13.989964 12919 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.991009 12963 raft_consensus.cc:357] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.991772 12963 raft_consensus.cc:383] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:13.992148 12963 raft_consensus.cc:738] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Initialized, Role: FOLLOWER
I20250623 01:57:13.992820 12963 consensus_queue.cc:260] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:13.993362 12844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:13.993953 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:13.994511 12920 raft_consensus.cc:3058] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:13.995540 12963 ts_tablet_manager.cc:1428] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250623 01:57:13.997103 12846 raft_consensus.cc:2466] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 1.
I20250623 01:57:13.998147 12737 leader_election.cc:304] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:13.998922 12973 raft_consensus.cc:2802] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:13.999428 12973 raft_consensus.cc:695] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:14.000190 12973 consensus_queue.cc:237] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.003480 12967 raft_consensus.cc:513] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.006760 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:14.007738 12846 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:14.009599 12919 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 1.
I20250623 01:57:14.010789 12967 leader_election.cc:290] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:14.011684 12918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:14.013082 12918 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:14.014667 12846 raft_consensus.cc:2466] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 1.
I20250623 01:57:14.015849 12737 leader_election.cc:304] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:14.016641 12967 raft_consensus.cc:2802] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:14.017242 12967 raft_consensus.cc:695] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:14.018062 12967 consensus_queue.cc:237] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.019675 12918 raft_consensus.cc:2466] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 1.
I20250623 01:57:14.011350 12920 raft_consensus.cc:2466] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 1.
I20250623 01:57:14.012393 13002 leader_election.cc:290] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:14.021978 12734 leader_election.cc:304] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters: 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:14.023101 13002 raft_consensus.cc:2802] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:14.023650 13002 raft_consensus.cc:695] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:14.024760 13002 consensus_queue.cc:237] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.027607 12667 catalog_manager.cc:5582] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 0 to 1, leader changed from <none> to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 1 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:14.044509 12667 catalog_manager.cc:5582] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 0 to 1, leader changed from <none> to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 1 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:14.046268 12667 catalog_manager.cc:5582] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 0 to 1, leader changed from <none> to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 1 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:14.069387 12996 raft_consensus.cc:491] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:14.069908 12996 raft_consensus.cc:513] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.072109 12996 leader_election.cc:290] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:14.072723 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df" is_pre_election: true
I20250623 01:57:14.073052 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:14.073336 12771 raft_consensus.cc:2466] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 0.
I20250623 01:57:14.073693 12846 raft_consensus.cc:2466] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 0.
I20250623 01:57:14.074872 12883 leader_election.cc:304] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:14.075722 12996 raft_consensus.cc:2802] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:14.076083 12996 raft_consensus.cc:491] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:14.076409 12996 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:14.157585 12996 raft_consensus.cc:513] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.161041 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:14.161571 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:14.161799 12771 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:14.162197 12846 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:14.162377 12996 leader_election.cc:290] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:14.167495 13002 raft_consensus.cc:491] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:14.167992 13002 raft_consensus.cc:513] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.169457 13002 leader_election.cc:290] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:14.170135 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08809c450cf0440dac8d4a2348b94d3c" is_pre_election: true
I20250623 01:57:14.170370 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:14.170881 12845 raft_consensus.cc:2391] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1941a8d154b24a47be40c5f39bba84df in current term 1: Already voted for candidate 8974a68eaebc4629b7ff34eeb07183bf in this term.
I20250623 01:57:14.172434 12737 leader_election.cc:304] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df; no voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:14.172973 13002 raft_consensus.cc:3058] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:14.176798 12981 raft_consensus.cc:491] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:14.177145 12981 raft_consensus.cc:513] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.178961 12981 leader_election.cc:290] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:14.180169 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf" is_pre_election: true
I20250623 01:57:14.180111 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1941a8d154b24a47be40c5f39bba84df" is_pre_election: true
I20250623 01:57:14.180819 12919 raft_consensus.cc:2391] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08809c450cf0440dac8d4a2348b94d3c in current term 1: Already voted for candidate 1941a8d154b24a47be40c5f39bba84df in this term.
I20250623 01:57:14.181676 12809 leader_election.cc:304] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c; no voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:14.182546 12981 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:14.190239 12967 consensus_queue.cc:1035] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.253777 12973 consensus_queue.cc:1035] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.260027 12771 raft_consensus.cc:2466] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 1.
I20250623 01:57:14.260166 12846 raft_consensus.cc:2466] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 1.
I20250623 01:57:14.266415 12886 leader_election.cc:304] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:14.267242 12996 raft_consensus.cc:2802] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:14.267681 12996 raft_consensus.cc:695] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:14.268458 12996 consensus_queue.cc:237] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.272686 12967 consensus_queue.cc:1035] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 01:57:14.287601 12716 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:57:14.293221 12667 catalog_manager.cc:5582] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 1 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:14.308708 12996 consensus_queue.cc:1035] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.319214 12972 consensus_queue.cc:1035] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.360658 13002 raft_consensus.cc:2747] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:14.360658 12981 raft_consensus.cc:2747] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:14.365830 13002 consensus_queue.cc:1035] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.365763 12972 consensus_queue.cc:1035] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.382236 12996 consensus_queue.cc:1035] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.413808 13002 consensus_queue.cc:1035] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.418519 13008 consensus_queue.cc:1035] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.426823 12973 consensus_queue.cc:1035] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:57:14.438108 12972 consensus_queue.cc:1035] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.507763 12845 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:14.530738 12973 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.532641 12973 consensus_queue.cc:1035] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.543594 12973 consensus_queue.cc:1035] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.568512 12973 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.589946 12919 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
mode: GRACEFUL
new_leader_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:33758
I20250623 01:57:14.590711 12919 raft_consensus.cc:604] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Received request to transfer leadership to TS 1941a8d154b24a47be40c5f39bba84df
I20250623 01:57:14.601135 12919 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
mode: GRACEFUL
new_leader_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:33758
I20250623 01:57:14.601634 12919 raft_consensus.cc:604] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Received request to transfer leadership to TS 1941a8d154b24a47be40c5f39bba84df
I20250623 01:57:14.620963 12771 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37702
I20250623 01:57:14.621598 12771 raft_consensus.cc:604] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:14.626410 12771 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37702
I20250623 01:57:14.626969 12771 raft_consensus.cc:604] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:14.631956 12919 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:33758
I20250623 01:57:14.632403 12919 raft_consensus.cc:604] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:14.636462 12919 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e737c71979a149d98198bcd4308ff967"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:33758
I20250623 01:57:14.637001 12919 raft_consensus.cc:604] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:14.640904 12771 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6093f636863d4e2fb12280c21a63be5a"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
mode: GRACEFUL
new_leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:37702
I20250623 01:57:14.641497 12771 raft_consensus.cc:604] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Received request to transfer leadership to TS 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:14.673137 12972 consensus_queue.cc:1035] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.680223 13008 raft_consensus.cc:991] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:14.680671 13008 raft_consensus.cc:1079] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:14.682996 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e737c71979a149d98198bcd4308ff967"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37708
I20250623 01:57:14.683534 12844 raft_consensus.cc:491] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:14.683923 12844 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:14.686957 13008 consensus_queue.cc:1035] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.773440 12844 raft_consensus.cc:513] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.774783 12845 raft_consensus.cc:1238] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Rejecting Update request from peer 8974a68eaebc4629b7ff34eeb07183bf for earlier term 1. Current term is 2. Ops: []
I20250623 01:57:14.775736 12844 leader_election.cc:290] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:14.777493 13008 consensus_queue.cc:1046] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:14.777967 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:14.777967 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:14.778343 13008 raft_consensus.cc:3053] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:14.778661 12770 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:14.778690 13008 raft_consensus.cc:738] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:14.779608 13008 consensus_queue.cc:260] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.780974 13008 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:14.827425 12996 raft_consensus.cc:991] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:14.827942 13045 raft_consensus.cc:1079] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:14.829568 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37708
I20250623 01:57:14.830111 12844 raft_consensus.cc:491] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:14.830456 12844 raft_consensus.cc:3058] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:14.870784 12770 raft_consensus.cc:2466] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:14.871209 12844 raft_consensus.cc:513] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.872586 12809 leader_election.cc:304] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:14.873761 12844 leader_election.cc:290] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:14.874392 12981 raft_consensus.cc:2802] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:14.876102 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:14.876955 12920 raft_consensus.cc:3053] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:14.877406 12920 raft_consensus.cc:738] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:14.878451 12920 consensus_queue.cc:260] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.880191 12920 raft_consensus.cc:3058] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:14.883011 12771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c40a8fa54e45b195056cb49339ac8f" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:14.883533 12771 raft_consensus.cc:3058] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:14.887125 12967 raft_consensus.cc:991] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df: : Instructing follower 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:14.887609 12967 raft_consensus.cc:1079] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Signalling peer 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:14.890213 12981 raft_consensus.cc:695] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:14.891007 12981 consensus_queue.cc:237] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.892098 12918 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6093f636863d4e2fb12280c21a63be5a"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:33738
I20250623 01:57:14.892666 12918 raft_consensus.cc:491] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:14.893040 12918 raft_consensus.cc:3058] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:14.899784 12666 catalog_manager.cc:5582] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 1 to 2, leader changed from 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 2 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:14.914660 13045 raft_consensus.cc:991] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf: : Instructing follower 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:14.915257 13045 raft_consensus.cc:1079] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Signalling peer 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:14.916908 12770 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:37688
I20250623 01:57:14.917410 12770 raft_consensus.cc:491] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:14.917676 12770 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:14.965731 12771 raft_consensus.cc:2466] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:14.966020 12919 raft_consensus.cc:2466] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:14.965732 12920 raft_consensus.cc:2466] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:14.966959 12809 leader_election.cc:304] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:14.968700 12981 raft_consensus.cc:2802] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:14.969203 12981 raft_consensus.cc:695] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:14.970067 12981 consensus_queue.cc:237] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:14.977373 12666 catalog_manager.cc:5582] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 1 to 2, leader changed from 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 2 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:14.984010 12973 raft_consensus.cc:991] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:14.984632 12973 raft_consensus.cc:1079] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:14.989787 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37698
I20250623 01:57:14.990363 12844 raft_consensus.cc:491] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:14.990715 12844 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.045645 13045 raft_consensus.cc:991] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf: : Instructing follower 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:15.046073 12996 raft_consensus.cc:1079] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Signalling peer 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:15.047317 12771 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:37688
I20250623 01:57:15.047763 12771 raft_consensus.cc:491] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:15.048033 12771 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.064234 12770 raft_consensus.cc:513] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.064234 12918 raft_consensus.cc:513] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.066743 12770 leader_election.cc:290] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 2 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:15.067739 12918 leader_election.cc:290] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 2 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:15.064236 12844 raft_consensus.cc:513] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.068512 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:15.069089 12845 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.068356 12846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6093f636863d4e2fb12280c21a63be5a" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:15.069640 12846 raft_consensus.cc:3058] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.069924 12844 leader_election.cc:290] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:15.071540 12769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:15.072113 12769 raft_consensus.cc:3053] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:15.072415 12769 raft_consensus.cc:738] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:15.072640 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6093f636863d4e2fb12280c21a63be5a" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:15.073217 12770 raft_consensus.cc:3053] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:15.072942 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:15.073608 12770 raft_consensus.cc:738] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:15.073400 12769 consensus_queue.cc:260] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.073992 12919 raft_consensus.cc:3053] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:15.074580 12770 consensus_queue.cc:260] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.075892 12770 raft_consensus.cc:3058] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.076792 12769 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.071907 12918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:15.077847 12918 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.075205 12919 raft_consensus.cc:738] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:15.080783 12919 consensus_queue.cc:260] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.081984 12919 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.137197 12967 raft_consensus.cc:991] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:15.137612 12973 raft_consensus.cc:1079] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:15.138790 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37698
I20250623 01:57:15.139178 12844 raft_consensus.cc:491] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:15.139449 12844 raft_consensus.cc:3058] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.167505 12845 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 2.
I20250623 01:57:15.168252 12846 raft_consensus.cc:2466] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 2.
I20250623 01:57:15.169361 12737 leader_election.cc:304] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:15.170464 12886 leader_election.cc:304] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:15.171437 12996 raft_consensus.cc:2802] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:15.171926 12996 raft_consensus.cc:695] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 2 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:15.170212 12771 raft_consensus.cc:513] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.172649 12996 consensus_queue.cc:237] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.173141 12973 raft_consensus.cc:2802] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:15.173628 12973 raft_consensus.cc:695] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:15.174654 12771 leader_election.cc:290] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 2 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:15.171994 12770 raft_consensus.cc:2466] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 2.
I20250623 01:57:15.174342 12973 consensus_queue.cc:237] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.176697 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:15.167551 12918 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:15.177263 12920 raft_consensus.cc:3053] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:15.177613 12920 raft_consensus.cc:738] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:15.178364 12920 consensus_queue.cc:260] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.178781 12809 leader_election.cc:304] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:15.179360 12920 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.179535 12981 raft_consensus.cc:2802] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:15.180644 12981 raft_consensus.cc:695] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:15.181505 12981 consensus_queue.cc:237] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.182482 12666 catalog_manager.cc:5582] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 1 to 2, leader changed from 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3) to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 2 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:15.183449 12845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:15.184027 12845 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.187912 12668 catalog_manager.cc:5582] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 1 to 2, leader changed from 1941a8d154b24a47be40c5f39bba84df (127.12.87.1) to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 2 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:15.190033 12665 catalog_manager.cc:5582] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 1 to 2, leader changed from 1941a8d154b24a47be40c5f39bba84df (127.12.87.1) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 2 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:15.269680 12769 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:15.269969 12920 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 2.
I20250623 01:57:15.269680 12919 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 2.
I20250623 01:57:15.271643 12734 leader_election.cc:304] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:15.269737 12844 raft_consensus.cc:513] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.272722 12973 raft_consensus.cc:2802] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:15.273221 12973 raft_consensus.cc:695] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:15.269973 12845 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 2.
I20250623 01:57:15.274060 12973 consensus_queue.cc:237] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.274583 12769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:15.274943 12844 leader_election.cc:290] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:15.275141 12769 raft_consensus.cc:3053] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:15.275467 12769 raft_consensus.cc:738] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:15.276121 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c79edf8d51584557b617641cdb0c5908" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:15.276166 12769 consensus_queue.cc:260] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.276896 12920 raft_consensus.cc:3058] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.277432 12769 raft_consensus.cc:3058] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:15.281828 12665 catalog_manager.cc:5582] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 1 to 2, leader changed from 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3) to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 2 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:15.317669 12919 raft_consensus.cc:1273] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.318770 12981 consensus_queue.cc:1035] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.328554 12770 raft_consensus.cc:1273] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.329985 13046 consensus_queue.cc:1035] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.369805 12919 raft_consensus.cc:1273] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.370345 12920 raft_consensus.cc:2466] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:15.370357 12769 raft_consensus.cc:2466] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:15.371690 12809 leader_election.cc:304] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:15.372323 13046 raft_consensus.cc:2802] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:15.372793 13052 consensus_queue.cc:1035] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.373099 13046 raft_consensus.cc:695] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:15.374158 13046 consensus_queue.cc:237] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:15.380123 12769 raft_consensus.cc:1273] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.381563 13046 consensus_queue.cc:1035] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.386214 12665 catalog_manager.cc:5582] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 1 to 2, leader changed from 1941a8d154b24a47be40c5f39bba84df (127.12.87.1) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 2 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:15.579728 12919 raft_consensus.cc:1273] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.580776 13046 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.589164 12769 raft_consensus.cc:1273] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.590627 13052 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.652848 12919 raft_consensus.cc:1273] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.653934 12973 consensus_queue.cc:1035] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.661576 12844 raft_consensus.cc:1273] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.662943 13059 consensus_queue.cc:1035] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.789564 12769 raft_consensus.cc:1273] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.790853 12996 consensus_queue.cc:1035] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:57:15.796754 12844 raft_consensus.cc:1273] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.800004 12972 consensus_queue.cc:1035] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.830538 12919 raft_consensus.cc:1273] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.831842 12920 raft_consensus.cc:1273] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.831816 13059 consensus_queue.cc:1035] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.833014 13052 consensus_queue.cc:1035] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.844839 12769 raft_consensus.cc:1273] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.845746 12844 raft_consensus.cc:1273] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:15.846736 13052 consensus_queue.cc:1035] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:15.847286 12973 consensus_queue.cc:1035] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:17.666931 12636 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20250623 01:57:17.667620 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:17.684891 12844 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
mode: GRACEFUL
new_leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:37718
I20250623 01:57:17.685412 12844 raft_consensus.cc:604] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Received request to transfer leadership to TS 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:17.694617 12769 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
mode: GRACEFUL
new_leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:37704
I20250623 01:57:17.695075 12769 raft_consensus.cc:604] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Received request to transfer leadership to TS 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:17.696122 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250623 01:57:17.696537 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:17.716755 12967 raft_consensus.cc:991] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df: : Instructing follower 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:17.717130 13073 raft_consensus.cc:1079] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Signalling peer 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:17.718371 12920 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:33738
I20250623 01:57:17.718773 12920 raft_consensus.cc:491] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:17.719036 12920 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:17.733597 12920 raft_consensus.cc:513] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:17.735097 12920 leader_election.cc:290] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 2 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:17.735915 12769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:17.736065 12844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:17.736557 12844 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:17.736688 12769 raft_consensus.cc:3053] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:17.737113 12769 raft_consensus.cc:738] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:17.737912 12769 consensus_queue.cc:260] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:17.739202 12769 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:17.740922 12844 raft_consensus.cc:2466] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 2.
I20250623 01:57:17.741868 12886 leader_election.cc:304] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:17.742611 13069 raft_consensus.cc:2802] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:17.743669 13069 raft_consensus.cc:695] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 2 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:17.744429 13069 consensus_queue.cc:237] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:17.750051 12666 catalog_manager.cc:5582] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 1 to 2, leader changed from 1941a8d154b24a47be40c5f39bba84df (127.12.87.1) to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 2 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:17.834861 12769 raft_consensus.cc:2466] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 2.
I20250623 01:57:17.860878 13052 raft_consensus.cc:991] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c: : Instructing follower 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:17.861231 13080 raft_consensus.cc:1079] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Signalling peer 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:17.862516 12920 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:33754
I20250623 01:57:17.862969 12920 raft_consensus.cc:491] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:17.863237 12920 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:17.934017 12920 raft_consensus.cc:513] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:17.936506 12920 leader_election.cc:290] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 3 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:17.937062 12844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:17.936820 12769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:17.937580 12844 raft_consensus.cc:3053] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:17.937752 12769 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:17.937944 12844 raft_consensus.cc:738] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:17.938743 12844 consensus_queue.cc:260] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:17.939921 12844 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:18.095230 12844 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 3.
I20250623 01:57:18.096083 12769 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 3.
I20250623 01:57:18.096495 12886 leader_election.cc:304] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:18.097625 13069 raft_consensus.cc:2802] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:18.098510 13069 raft_consensus.cc:695] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:18.099619 13069 consensus_queue.cc:237] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:18.136276 12844 raft_consensus.cc:1273] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:18.138525 13069 consensus_queue.cc:1035] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:57:18.159538 12769 raft_consensus.cc:1273] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:18.160801 13069 consensus_queue.cc:1035] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:18.178542 12666 catalog_manager.cc:5582] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 2 to 3, leader changed from 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2) to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 3 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:18.683655 12844 raft_consensus.cc:1273] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:18.684790 13074 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:18.698344 12769 raft_consensus.cc:1273] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:18.699753 13088 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:18.715970 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:18.720423 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:18.720760 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250623 01:57:19.012446 12699 debug-util.cc:398] Leaking SignalData structure 0x7b0800061060 after lost signal to thread 12639
W20250623 01:57:19.013141 12699 debug-util.cc:398] Leaking SignalData structure 0x7b0800061740 after lost signal to thread 12702
W20250623 01:57:19.013895 12699 debug-util.cc:398] Leaking SignalData structure 0x7b08000409e0 after lost signal to thread 12795
W20250623 01:57:19.014566 12699 debug-util.cc:398] Leaking SignalData structure 0x7b08001c5a40 after lost signal to thread 12870
W20250623 01:57:19.015199 12699 debug-util.cc:398] Leaking SignalData structure 0x7b080019b4e0 after lost signal to thread 12944
I20250623 01:57:19.721549 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:19.726163 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:19.726537 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:20.727309 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:20.732260 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:20.732700 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:21.733805 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:21.738598 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:21.738932 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:22.739833 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:22.744262 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:22.744575 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:23.745430 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:23.751039 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:23.751384 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:24.752094 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:24.756385 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:24.756717 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:25.757572 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:25.762646 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:25.763129 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:26.764088 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:26.769968 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:26.770362 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:27.794251 12636 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20250623 01:57:27.798214 12770 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37702
I20250623 01:57:27.798861 12770 raft_consensus.cc:604] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:27.803081 12770 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37702
I20250623 01:57:27.803527 12770 raft_consensus.cc:604] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:27.807638 12920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:33758
I20250623 01:57:27.808040 12920 raft_consensus.cc:604] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 2 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:27.812101 12770 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37702
I20250623 01:57:27.812558 12770 raft_consensus.cc:604] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:27.818406 12920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
mode: GRACEFUL
new_leader_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:33758
I20250623 01:57:27.818809 12920 raft_consensus.cc:604] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 LEADER]: Received request to transfer leadership to TS 08809c450cf0440dac8d4a2348b94d3c
I20250623 01:57:27.914047 13101 raft_consensus.cc:991] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:27.914562 13102 raft_consensus.cc:1079] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:27.915900 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37698
I20250623 01:57:27.916288 12844 raft_consensus.cc:491] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:27.916564 12844 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:27.920912 12844 raft_consensus.cc:513] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:27.922286 12844 leader_election.cc:290] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 3 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:27.923244 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:27.923151 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:27.923776 12920 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:27.923914 12770 raft_consensus.cc:3053] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:27.924280 12770 raft_consensus.cc:738] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:27.925027 12770 consensus_queue.cc:260] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:27.926126 12770 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:27.928637 12920 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 3.
I20250623 01:57:27.929863 13101 raft_consensus.cc:991] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:27.929833 12809 leader_election.cc:304] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:27.930375 13102 raft_consensus.cc:1079] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:27.930956 13080 raft_consensus.cc:2802] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:27.931844 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37698
I20250623 01:57:27.932174 12770 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 3.
I20250623 01:57:27.932528 12844 raft_consensus.cc:491] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:27.933018 12844 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:27.933429 13080 raft_consensus.cc:695] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:27.934231 13080 consensus_queue.cc:237] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:27.938746 12844 raft_consensus.cc:513] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:27.940171 12844 leader_election.cc:290] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 3 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:27.941032 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:27.941268 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:27.941624 12770 raft_consensus.cc:3053] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:27.941735 12920 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:27.941948 12770 raft_consensus.cc:738] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:27.942704 12770 consensus_queue.cc:260] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:27.944012 12770 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:27.944216 12666 catalog_manager.cc:5582] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 2 to 3, leader changed from 1941a8d154b24a47be40c5f39bba84df (127.12.87.1) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 3 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:27.947971 12920 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 3.
I20250623 01:57:27.949039 12809 leader_election.cc:304] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:27.949867 13080 raft_consensus.cc:2802] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:27.950322 13080 raft_consensus.cc:695] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:27.950642 12770 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 3.
I20250623 01:57:27.951278 13080 consensus_queue.cc:237] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:27.961830 12666 catalog_manager.cc:5582] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 2 to 3, leader changed from 1941a8d154b24a47be40c5f39bba84df (127.12.87.1) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 3 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:28.233698 13098 raft_consensus.cc:991] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:28.234162 13110 raft_consensus.cc:1079] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 2 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:28.235405 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37708
I20250623 01:57:28.235803 12844 raft_consensus.cc:491] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:28.236076 12844 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:28.240331 12844 raft_consensus.cc:513] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.241648 12844 leader_election.cc:290] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 3 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:28.242631 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:28.242769 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c52d16a94d354ba290e8e7e4335a548b" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:28.243208 12770 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:28.243259 12920 raft_consensus.cc:3053] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:28.243701 12920 raft_consensus.cc:738] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:28.244410 12920 consensus_queue.cc:260] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.245563 12920 raft_consensus.cc:3058] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:28.249825 12770 raft_consensus.cc:2466] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 3.
I20250623 01:57:28.250922 12809 leader_election.cc:304] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:28.251333 12920 raft_consensus.cc:2466] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 3.
I20250623 01:57:28.251549 13080 raft_consensus.cc:2802] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:28.251991 13080 raft_consensus.cc:695] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:28.252604 13080 consensus_queue.cc:237] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.258854 12666 catalog_manager.cc:5582] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 2 to 3, leader changed from 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 3 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:28.269519 13101 raft_consensus.cc:991] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:28.269871 13102 raft_consensus.cc:1079] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:28.271059 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37698
I20250623 01:57:28.271435 12844 raft_consensus.cc:491] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:28.271667 12844 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:28.275430 12844 raft_consensus.cc:513] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.276607 12844 leader_election.cc:290] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:28.277356 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:28.277532 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:28.277791 12770 raft_consensus.cc:3053] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:28.277940 12920 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:28.278091 12770 raft_consensus.cc:738] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:28.278820 12770 consensus_queue.cc:260] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.279745 12770 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:28.282724 13098 raft_consensus.cc:991] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf: : Instructing follower 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:28.282748 12920 raft_consensus.cc:2466] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:28.283304 13098 raft_consensus.cc:1079] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 LEADER]: Signalling peer 08809c450cf0440dac8d4a2348b94d3c to start an election
I20250623 01:57:28.283813 12809 leader_election.cc:304] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:28.284727 13080 raft_consensus.cc:2802] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:28.284816 12844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
from {username='slave'} at 127.0.0.1:37708
I20250623 01:57:28.285171 12770 raft_consensus.cc:2466] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 2.
I20250623 01:57:28.285236 13080 raft_consensus.cc:695] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:28.285336 12844 raft_consensus.cc:491] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:28.286103 12844 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:28.286221 13080 consensus_queue.cc:237] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.290537 12844 raft_consensus.cc:513] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.292186 12844 leader_election.cc:290] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 4 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:28.292940 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:28.293459 12770 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:28.293608 12920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "08809c450cf0440dac8d4a2348b94d3c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:28.294090 12920 raft_consensus.cc:3053] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:57:28.294363 12920 raft_consensus.cc:738] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:28.293817 12666 catalog_manager.cc:5582] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 1 to 2, leader changed from 1941a8d154b24a47be40c5f39bba84df (127.12.87.1) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 2 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:28.294952 12920 consensus_queue.cc:260] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.295931 12920 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:28.298579 12770 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 4.
I20250623 01:57:28.299446 12809 leader_election.cc:304] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08809c450cf0440dac8d4a2348b94d3c, 1941a8d154b24a47be40c5f39bba84df; no voters:
I20250623 01:57:28.300179 13080 raft_consensus.cc:2802] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:57:28.300511 13080 raft_consensus.cc:695] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 4 LEADER]: Becoming Leader. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:28.300678 12920 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08809c450cf0440dac8d4a2348b94d3c in term 4.
I20250623 01:57:28.301119 13080 consensus_queue.cc:237] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:28.307909 12666 catalog_manager.cc:5582] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c reported cstate change: term changed from 3 to 4, leader changed from 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3) to 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2). New cstate: current_term: 4 leader_uuid: "08809c450cf0440dac8d4a2348b94d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:28.370851 12920 raft_consensus.cc:1273] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:28.372009 13080 consensus_queue.cc:1035] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:28.382570 12770 raft_consensus.cc:1273] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:28.384423 13103 consensus_queue.cc:1035] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:57:28.501900 12920 raft_consensus.cc:1273] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:28.503055 12770 raft_consensus.cc:1273] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:28.502995 13103 consensus_queue.cc:1035] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:28.504280 13103 consensus_queue.cc:1035] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:28.736057 12920 raft_consensus.cc:1273] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:28.737156 13103 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:57:28.746511 12770 raft_consensus.cc:1273] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:28.748204 13103 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:57:28.754449 12920 raft_consensus.cc:1273] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:28.755702 13103 consensus_queue.cc:1035] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:28.765910 12770 raft_consensus.cc:1273] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:28.770522 13103 consensus_queue.cc:1035] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:57:28.799003 12920 raft_consensus.cc:1273] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:28.800060 13103 consensus_queue.cc:1035] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:28.808687 12770 raft_consensus.cc:1273] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Refusing update from remote peer 08809c450cf0440dac8d4a2348b94d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:28.810187 13127 consensus_queue.cc:1035] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:30.850244 12636 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20250623 01:57:30.850829 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:30.857005 12844 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
mode: GRACEFUL
new_leader_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:37718
I20250623 01:57:30.857468 12844 raft_consensus.cc:604] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Received request to transfer leadership to TS 1941a8d154b24a47be40c5f39bba84df
I20250623 01:57:30.859350 12844 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
mode: GRACEFUL
new_leader_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:37718
I20250623 01:57:30.859681 12844 raft_consensus.cc:604] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 4 LEADER]: Received request to transfer leadership to TS 1941a8d154b24a47be40c5f39bba84df
I20250623 01:57:30.861426 12844 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
mode: GRACEFUL
new_leader_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:37718
I20250623 01:57:30.861773 12844 raft_consensus.cc:604] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Received request to transfer leadership to TS 1941a8d154b24a47be40c5f39bba84df
I20250623 01:57:30.863472 12844 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
mode: GRACEFUL
new_leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:37718
I20250623 01:57:30.863823 12844 raft_consensus.cc:604] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Received request to transfer leadership to TS 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:30.865520 12844 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e737c71979a149d98198bcd4308ff967"
dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
mode: GRACEFUL
new_leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:37718
I20250623 01:57:30.865850 12844 raft_consensus.cc:604] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Received request to transfer leadership to TS 8974a68eaebc4629b7ff34eeb07183bf
I20250623 01:57:30.866561 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:57:30.866916 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:30.890309 13116 raft_consensus.cc:991] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c: : Instructing follower 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:30.890699 13129 raft_consensus.cc:1079] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Signalling peer 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:30.892028 12919 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:33754
I20250623 01:57:30.892421 12919 raft_consensus.cc:491] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:30.892652 12919 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:30.896543 12919 raft_consensus.cc:513] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:30.898190 12919 leader_election.cc:290] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 4 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427)
I20250623 01:57:30.898974 12844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:30.899142 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8034e2a5ed434721b58a76f8d8310e33" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:30.899390 12844 raft_consensus.cc:3053] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:57:30.899580 12770 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:30.899696 12844 raft_consensus.cc:738] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:30.900332 12844 consensus_queue.cc:260] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:30.901109 12844 raft_consensus.cc:3058] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:30.903359 12770 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 4.
I20250623 01:57:30.904271 12883 leader_election.cc:304] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:30.904925 13110 raft_consensus.cc:2802] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:57:30.905042 12844 raft_consensus.cc:2466] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 4.
I20250623 01:57:30.905941 13110 raft_consensus.cc:695] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:30.906755 13110 consensus_queue.cc:237] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:30.912773 12665 catalog_manager.cc:5582] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 3 to 4, leader changed from 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2) to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 4 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:30.970353 13116 raft_consensus.cc:991] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c: : Instructing follower 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:30.971051 13129 raft_consensus.cc:1079] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 4 LEADER]: Signalling peer 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:30.972798 12770 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:37674
I20250623 01:57:30.973364 12770 raft_consensus.cc:491] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:30.973637 12770 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:57:30.978677 12770 raft_consensus.cc:513] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:30.980641 12770 leader_election.cc:290] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 5 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:30.982445 12844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:30.982470 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a83d61adce49d8ab74a9f4ddab4cf3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:30.983052 12844 raft_consensus.cc:3053] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:57:30.983213 12919 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:57:30.983314 12844 raft_consensus.cc:738] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:30.984107 12844 consensus_queue.cc:260] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:30.988976 12919 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 5.
I20250623 01:57:30.990197 12844 raft_consensus.cc:3058] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:57:30.990506 12734 leader_election.cc:304] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:30.992304 13137 raft_consensus.cc:2802] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:57:30.993458 13137 raft_consensus.cc:695] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 5 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:30.994372 13137 consensus_queue.cc:237] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:30.995585 12844 raft_consensus.cc:2466] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 5.
I20250623 01:57:31.001678 12665 catalog_manager.cc:5582] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 4 to 5, leader changed from 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2) to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 5 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:31.066318 13116 raft_consensus.cc:991] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c: : Instructing follower 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:31.066830 13129 raft_consensus.cc:1079] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Signalling peer 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:31.068178 12770 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:37674
I20250623 01:57:31.068562 12770 raft_consensus.cc:491] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:31.068796 12770 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:31.072826 12770 raft_consensus.cc:513] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.074016 12770 leader_election.cc:290] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 4 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:31.074842 12844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:31.074956 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5442a9db195b424a8cbb938180e80a45" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:31.075367 12844 raft_consensus.cc:3053] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:57:31.075443 12919 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:31.075753 12844 raft_consensus.cc:738] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:31.076509 12844 consensus_queue.cc:260] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.077713 12844 raft_consensus.cc:3058] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:31.081233 12919 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 4.
I20250623 01:57:31.082122 12734 leader_election.cc:304] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:31.082746 13137 raft_consensus.cc:2802] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:57:31.082957 12844 raft_consensus.cc:2466] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 4.
I20250623 01:57:31.083088 13137 raft_consensus.cc:695] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 4 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:31.083986 13137 consensus_queue.cc:237] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.090334 12665 catalog_manager.cc:5582] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 3 to 4, leader changed from 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2) to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 4 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:31.106268 13116 raft_consensus.cc:991] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c: : Instructing follower 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:31.106683 13129 raft_consensus.cc:1079] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Signalling peer 8974a68eaebc4629b7ff34eeb07183bf to start an election
I20250623 01:57:31.107932 12919 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e737c71979a149d98198bcd4308ff967"
dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
from {username='slave'} at 127.0.0.1:33754
I20250623 01:57:31.108314 12919 raft_consensus.cc:491] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:31.108572 12919 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:31.112469 12919 raft_consensus.cc:513] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.113757 12919 leader_election.cc:290] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 3 election: Requested vote from peers 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427), 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703)
I20250623 01:57:31.114553 12770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
I20250623 01:57:31.114707 12844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e737c71979a149d98198bcd4308ff967" candidate_uuid: "8974a68eaebc4629b7ff34eeb07183bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:31.115092 12770 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:31.115123 12844 raft_consensus.cc:3053] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:31.115564 12844 raft_consensus.cc:738] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:31.116209 12844 consensus_queue.cc:260] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.117276 12844 raft_consensus.cc:3058] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:31.120829 12770 raft_consensus.cc:2466] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 3.
I20250623 01:57:31.121949 12883 leader_election.cc:304] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:31.122507 12844 raft_consensus.cc:2466] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8974a68eaebc4629b7ff34eeb07183bf in term 3.
I20250623 01:57:31.122664 13110 raft_consensus.cc:2802] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:31.123123 13110 raft_consensus.cc:695] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 LEADER]: Becoming Leader. State: Replica: 8974a68eaebc4629b7ff34eeb07183bf, State: Running, Role: LEADER
I20250623 01:57:31.123920 13110 consensus_queue.cc:237] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.130180 12665 catalog_manager.cc:5582] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf reported cstate change: term changed from 2 to 3, leader changed from 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2) to 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3). New cstate: current_term: 3 leader_uuid: "8974a68eaebc4629b7ff34eeb07183bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:31.157615 13116 raft_consensus.cc:991] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c: : Instructing follower 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:31.158069 13129 raft_consensus.cc:1079] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Signalling peer 1941a8d154b24a47be40c5f39bba84df to start an election
I20250623 01:57:31.159384 12770 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3"
dest_uuid: "1941a8d154b24a47be40c5f39bba84df"
from {username='slave'} at 127.0.0.1:37674
I20250623 01:57:31.159760 12770 raft_consensus.cc:491] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:31.160012 12770 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:31.164414 12770 raft_consensus.cc:513] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.165613 12770 leader_election.cc:290] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 3 election: Requested vote from peers 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703), 8974a68eaebc4629b7ff34eeb07183bf (127.12.87.3:34451)
I20250623 01:57:31.166330 12844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "08809c450cf0440dac8d4a2348b94d3c"
I20250623 01:57:31.166582 12919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7177b120b41a44e386b315897a02e2b3" candidate_uuid: "1941a8d154b24a47be40c5f39bba84df" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8974a68eaebc4629b7ff34eeb07183bf"
I20250623 01:57:31.166846 12844 raft_consensus.cc:3053] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:31.167117 12919 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:31.167153 12844 raft_consensus.cc:738] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 08809c450cf0440dac8d4a2348b94d3c, State: Running, Role: LEADER
I20250623 01:57:31.167979 12844 consensus_queue.cc:260] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.169163 12844 raft_consensus.cc:3058] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:31.172983 12919 raft_consensus.cc:2466] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 3.
I20250623 01:57:31.173965 12734 leader_election.cc:304] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1941a8d154b24a47be40c5f39bba84df, 8974a68eaebc4629b7ff34eeb07183bf; no voters:
I20250623 01:57:31.174485 12844 raft_consensus.cc:2466] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1941a8d154b24a47be40c5f39bba84df in term 3.
I20250623 01:57:31.174618 13137 raft_consensus.cc:2802] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:31.175036 13137 raft_consensus.cc:695] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 3 LEADER]: Becoming Leader. State: Replica: 1941a8d154b24a47be40c5f39bba84df, State: Running, Role: LEADER
I20250623 01:57:31.175822 13137 consensus_queue.cc:237] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } }
I20250623 01:57:31.182126 12665 catalog_manager.cc:5582] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df reported cstate change: term changed from 2 to 3, leader changed from 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2) to 1941a8d154b24a47be40c5f39bba84df (127.12.87.1). New cstate: current_term: 3 leader_uuid: "1941a8d154b24a47be40c5f39bba84df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:31.391220 12919 raft_consensus.cc:1273] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 5 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:57:31.392233 13137 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:57:31.402107 12844 raft_consensus.cc:1273] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 5 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:57:31.403313 13137 consensus_queue.cc:1035] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:57:31.420646 12844 raft_consensus.cc:1273] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:31.421674 13110 consensus_queue.cc:1035] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:57:31.429716 12769 raft_consensus.cc:1273] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:31.430912 13110 consensus_queue.cc:1035] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:57:31.463580 12844 raft_consensus.cc:1273] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:31.464485 13137 consensus_queue.cc:1035] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:57:31.472167 12919 raft_consensus.cc:1273] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:31.473778 13151 consensus_queue.cc:1035] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:57:31.540589 12769 raft_consensus.cc:1273] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:31.541877 13110 consensus_queue.cc:1035] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1941a8d154b24a47be40c5f39bba84df" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:31.549607 12844 raft_consensus.cc:1273] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Refusing update from remote peer 8974a68eaebc4629b7ff34eeb07183bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:31.550925 13110 consensus_queue.cc:1035] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:31.710839 12919 raft_consensus.cc:1273] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:31.711869 13151 consensus_queue.cc:1035] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "8974a68eaebc4629b7ff34eeb07183bf" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:31.720593 12844 raft_consensus.cc:1273] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Refusing update from remote peer 1941a8d154b24a47be40c5f39bba84df: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:31.722034 13138 consensus_queue.cc:1035] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [LEADER]: Connected to new peer: Peer: permanent_uuid: "08809c450cf0440dac8d4a2348b94d3c" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:31.868108 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:31.873152 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:31.873642 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:32.874958 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:32.879510 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:32.879911 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:33.880944 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:33.885939 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:33.886348 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:34.887286 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:34.891956 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:34.892362 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:35.893500 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:35.898069 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:35.898496 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:36.899485 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:36.904091 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:36.904469 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:37.905514 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:37.911489 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:37.911886 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:39.538116 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:39.542712 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:39.543020 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250623 01:57:39.893735 12867 debug-util.cc:398] Leaking SignalData structure 0x7b080012e060 after lost signal to thread 12639
W20250623 01:57:39.894457 12867 debug-util.cc:398] Leaking SignalData structure 0x7b080012e740 after lost signal to thread 12702
W20250623 01:57:39.895144 12867 debug-util.cc:398] Leaking SignalData structure 0x7b08001bcc80 after lost signal to thread 12795
W20250623 01:57:39.895805 12867 debug-util.cc:398] Leaking SignalData structure 0x7b08001e7a00 after lost signal to thread 12870
W20250623 01:57:39.896420 12867 debug-util.cc:398] Leaking SignalData structure 0x7b08001e5540 after lost signal to thread 12944
I20250623 01:57:40.544080 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:40.551537 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:40.552022 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:41.579519 12636 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20250623 01:57:41.580168 12636 tablet_server.cc:178] TabletServer@127.12.87.1:0 shutting down...
W20250623 01:57:41.617516 12809 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250623 01:57:41.621624 12809 consensus_peers.cc:487] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c -> Peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Couldn't send request to peer 1941a8d154b24a47be40c5f39bba84df. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:57:41.621819 12636 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250623 01:57:41.622666 12636 tablet_replica.cc:331] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.623510 12636 raft_consensus.cc:2241] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.624200 12636 raft_consensus.cc:2270] T b6c40a8fa54e45b195056cb49339ac8f P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.626672 12636 tablet_replica.cc:331] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.627202 12636 raft_consensus.cc:2241] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.627748 12636 raft_consensus.cc:2270] T c79edf8d51584557b617641cdb0c5908 P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.629528 12636 tablet_replica.cc:331] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.630000 12636 raft_consensus.cc:2241] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.630525 12636 raft_consensus.cc:2270] T 6093f636863d4e2fb12280c21a63be5a P 1941a8d154b24a47be40c5f39bba84df [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.632367 12636 tablet_replica.cc:331] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.632866 12636 raft_consensus.cc:2241] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.633435 12636 raft_consensus.cc:2270] T c52d16a94d354ba290e8e7e4335a548b P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.635138 12636 tablet_replica.cc:331] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.635625 12636 raft_consensus.cc:2241] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 5 LEADER]: Raft consensus shutting down.
I20250623 01:57:41.636412 12636 raft_consensus.cc:2270] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 1941a8d154b24a47be40c5f39bba84df [term 5 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.638108 12636 tablet_replica.cc:331] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.638716 12636 raft_consensus.cc:2241] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.639235 12636 raft_consensus.cc:2270] T 8034e2a5ed434721b58a76f8d8310e33 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.641407 12636 tablet_replica.cc:331] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.641873 12636 raft_consensus.cc:2241] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.642321 12636 raft_consensus.cc:2270] T e737c71979a149d98198bcd4308ff967 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.643852 12636 tablet_replica.cc:331] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.644284 12636 raft_consensus.cc:2241] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 4 LEADER]: Raft consensus shutting down.
I20250623 01:57:41.644994 12636 raft_consensus.cc:2270] T 5442a9db195b424a8cbb938180e80a45 P 1941a8d154b24a47be40c5f39bba84df [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.646624 12636 tablet_replica.cc:331] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df: stopping tablet replica
I20250623 01:57:41.647027 12636 raft_consensus.cc:2241] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 3 LEADER]: Raft consensus shutting down.
I20250623 01:57:41.647712 12636 raft_consensus.cc:2270] T 7177b120b41a44e386b315897a02e2b3 P 1941a8d154b24a47be40c5f39bba84df [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.753116 12636 tablet_server.cc:195] TabletServer@127.12.87.1:0 shutdown complete.
W20250623 01:57:41.755605 12809 consensus_peers.cc:487] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c -> Peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Couldn't send request to peer 1941a8d154b24a47be40c5f39bba84df. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34427: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 01:57:41.783149 12636 tablet_server.cc:178] TabletServer@127.12.87.2:0 shutting down...
W20250623 01:57:41.795245 12886 consensus_peers.cc:487] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf -> Peer 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703): Couldn't send request to peer 08809c450cf0440dac8d4a2348b94d3c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:57:41.822402 12636 ts_tablet_manager.cc:1500] Shutting down tablet manager...
W20250623 01:57:41.822422 12886 consensus_peers.cc:487] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf -> Peer 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703): Couldn't send request to peer 08809c450cf0440dac8d4a2348b94d3c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:57:41.823329 12636 tablet_replica.cc:331] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.824327 12636 raft_consensus.cc:2241] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.824937 12636 raft_consensus.cc:2270] T 7177b120b41a44e386b315897a02e2b3 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.827536 12636 tablet_replica.cc:331] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.828083 12636 raft_consensus.cc:2241] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 3 LEADER]: Raft consensus shutting down.
I20250623 01:57:41.828856 12636 raft_consensus.cc:2270] T c52d16a94d354ba290e8e7e4335a548b P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.830544 12636 tablet_replica.cc:331] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.830999 12636 raft_consensus.cc:2241] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Raft consensus shutting down.
I20250623 01:57:41.831784 12636 raft_consensus.cc:2270] T b6c40a8fa54e45b195056cb49339ac8f P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.833402 12636 tablet_replica.cc:331] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.833889 12636 raft_consensus.cc:2241] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.834293 12636 raft_consensus.cc:2270] T 6093f636863d4e2fb12280c21a63be5a P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.835867 12636 tablet_replica.cc:331] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.836323 12636 raft_consensus.cc:2241] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.836707 12636 raft_consensus.cc:2270] T e737c71979a149d98198bcd4308ff967 P 08809c450cf0440dac8d4a2348b94d3c [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.838454 12636 tablet_replica.cc:331] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.838882 12636 raft_consensus.cc:2241] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.839267 12636 raft_consensus.cc:2270] T 5442a9db195b424a8cbb938180e80a45 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.840834 12636 tablet_replica.cc:331] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.841259 12636 raft_consensus.cc:2241] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 2 LEADER]: Raft consensus shutting down.
I20250623 01:57:41.842024 12636 raft_consensus.cc:2270] T c79edf8d51584557b617641cdb0c5908 P 08809c450cf0440dac8d4a2348b94d3c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.843673 12636 tablet_replica.cc:331] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.844108 12636 raft_consensus.cc:2241] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 5 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.844501 12636 raft_consensus.cc:2270] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 08809c450cf0440dac8d4a2348b94d3c [term 5 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:41.846156 12636 tablet_replica.cc:331] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c: stopping tablet replica
I20250623 01:57:41.846557 12636 raft_consensus.cc:2241] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:41.846922 12636 raft_consensus.cc:2270] T 8034e2a5ed434721b58a76f8d8310e33 P 08809c450cf0440dac8d4a2348b94d3c [term 4 FOLLOWER]: Raft consensus is shut down!
W20250623 01:57:41.999617 12883 consensus_peers.cc:487] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf -> Peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Couldn't send request to peer 1941a8d154b24a47be40c5f39bba84df. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34427: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 01:57:42.009748 12883 consensus_peers.cc:487] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf -> Peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Couldn't send request to peer 1941a8d154b24a47be40c5f39bba84df. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34427: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 01:57:42.077090 12886 consensus_peers.cc:487] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf -> Peer 08809c450cf0440dac8d4a2348b94d3c (127.12.87.2:39703): Couldn't send request to peer 08809c450cf0440dac8d4a2348b94d3c. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.2:39703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 01:57:42.110106 12636 tablet_server.cc:195] TabletServer@127.12.87.2:0 shutdown complete.
I20250623 01:57:42.126273 12636 tablet_server.cc:178] TabletServer@127.12.87.3:0 shutting down...
I20250623 01:57:42.150288 12636 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250623 01:57:42.151052 12636 tablet_replica.cc:331] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
W20250623 01:57:42.151677 12883 consensus_peers.cc:487] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf -> Peer 1941a8d154b24a47be40c5f39bba84df (127.12.87.1:34427): Couldn't send request to peer 1941a8d154b24a47be40c5f39bba84df. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34427: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 01:57:42.151877 12636 raft_consensus.cc:2241] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:42.152374 12636 raft_consensus.cc:2270] T c79edf8d51584557b617641cdb0c5908 P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.153968 12636 tablet_replica.cc:331] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
I20250623 01:57:42.154430 12636 raft_consensus.cc:2241] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:42.154814 12636 raft_consensus.cc:2270] T c52d16a94d354ba290e8e7e4335a548b P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.156339 12636 tablet_replica.cc:331] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
I20250623 01:57:42.156813 12636 raft_consensus.cc:2241] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 2 LEADER]: Raft consensus shutting down.
I20250623 01:57:42.157495 12636 raft_consensus.cc:2270] T 6093f636863d4e2fb12280c21a63be5a P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.159044 12636 tablet_replica.cc:331] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
I20250623 01:57:42.159485 12636 raft_consensus.cc:2241] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:42.159878 12636 raft_consensus.cc:2270] T 5442a9db195b424a8cbb938180e80a45 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.161484 12636 tablet_replica.cc:331] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
I20250623 01:57:42.161914 12636 raft_consensus.cc:2241] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 LEADER]: Raft consensus shutting down.
I20250623 01:57:42.162583 12636 raft_consensus.cc:2270] T e737c71979a149d98198bcd4308ff967 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.164014 12636 tablet_replica.cc:331] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
I20250623 01:57:42.164469 12636 raft_consensus.cc:2241] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 5 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:42.164834 12636 raft_consensus.cc:2270] T 59a83d61adce49d8ab74a9f4ddab4cf3 P 8974a68eaebc4629b7ff34eeb07183bf [term 5 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.166333 12636 tablet_replica.cc:331] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
I20250623 01:57:42.166764 12636 raft_consensus.cc:2241] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 LEADER]: Raft consensus shutting down.
I20250623 01:57:42.167388 12636 raft_consensus.cc:2270] T 8034e2a5ed434721b58a76f8d8310e33 P 8974a68eaebc4629b7ff34eeb07183bf [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.168860 12636 tablet_replica.cc:331] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
I20250623 01:57:42.169283 12636 raft_consensus.cc:2241] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:42.169714 12636 raft_consensus.cc:2270] T 7177b120b41a44e386b315897a02e2b3 P 8974a68eaebc4629b7ff34eeb07183bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.171134 12636 tablet_replica.cc:331] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf: stopping tablet replica
I20250623 01:57:42.171504 12636 raft_consensus.cc:2241] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:57:42.171833 12636 raft_consensus.cc:2270] T b6c40a8fa54e45b195056cb49339ac8f P 8974a68eaebc4629b7ff34eeb07183bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.197623 12636 tablet_server.cc:195] TabletServer@127.12.87.3:0 shutdown complete.
I20250623 01:57:42.213567 12636 master.cc:561] Master@127.12.87.62:36591 shutting down...
I20250623 01:57:42.401211 12636 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 1 LEADER]: Raft consensus shutting down.
I20250623 01:57:42.401824 12636 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:57:42.402196 12636 tablet_replica.cc:331] T 00000000000000000000000000000000 P 4fe5f9e622af439fba9683f9c1e27387: stopping tablet replica
I20250623 01:57:42.439682 12636 master.cc:583] Master@127.12.87.62:36591 shutdown complete.
08809c450cf0440dac8d4a2348b94d3c 4
1941a8d154b24a47be40c5f39bba84df 4
8974a68eaebc4629b7ff34eeb07183bf 1
08809c450cf0440dac8d4a2348b94d3c 3
1941a8d154b24a47be40c5f39bba84df 3
8974a68eaebc4629b7ff34eeb07183bf 3
08809c450cf0440dac8d4a2348b94d3c 8
1941a8d154b24a47be40c5f39bba84df 0
8974a68eaebc4629b7ff34eeb07183bf 1
08809c450cf0440dac8d4a2348b94d3c 3
1941a8d154b24a47be40c5f39bba84df 3
8974a68eaebc4629b7ff34eeb07183bf 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31616 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250623 01:57:42.478530 12636 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.87.62:45311
I20250623 01:57:42.479614 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:42.484362 13177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:42.485456 13178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:42.485757 13180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:42.486016 12636 server_base.cc:1048] running on GCE node
I20250623 01:57:42.486881 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:57:42.487061 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:57:42.487200 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643862487184 us; error 0 us; skew 500 ppm
I20250623 01:57:42.487627 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:42.490145 12636 webserver.cc:469] Webserver started at http://127.12.87.62:33531/ using document root <none> and password file <none>
I20250623 01:57:42.490610 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:42.490774 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:42.491029 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:42.492148 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/master-0-root/instance:
uuid: "a4a0ff7480744f68b50ddd187eb4351d"
format_stamp: "Formatted at 2025-06-23 01:57:42 on dist-test-slave-47g2"
I20250623 01:57:42.496207 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250623 01:57:42.499018 13185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.499760 12636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250623 01:57:42.499982 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/master-0-root
uuid: "a4a0ff7480744f68b50ddd187eb4351d"
format_stamp: "Formatted at 2025-06-23 01:57:42 on dist-test-slave-47g2"
I20250623 01:57:42.500205 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:42.515336 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:42.516436 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:42.551682 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.62:45311
I20250623 01:57:42.551774 13236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.62:45311 every 8 connection(s)
I20250623 01:57:42.555512 13237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:42.564714 13237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d: Bootstrap starting.
I20250623 01:57:42.569190 13237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:42.572939 13237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d: No bootstrap required, opened a new log
I20250623 01:57:42.574921 13237 raft_consensus.cc:357] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a0ff7480744f68b50ddd187eb4351d" member_type: VOTER }
I20250623 01:57:42.575333 13237 raft_consensus.cc:383] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:42.575538 13237 raft_consensus.cc:738] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a0ff7480744f68b50ddd187eb4351d, State: Initialized, Role: FOLLOWER
I20250623 01:57:42.576030 13237 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a0ff7480744f68b50ddd187eb4351d" member_type: VOTER }
I20250623 01:57:42.576462 13237 raft_consensus.cc:397] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:57:42.576674 13237 raft_consensus.cc:491] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:57:42.576931 13237 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:42.580745 13237 raft_consensus.cc:513] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a0ff7480744f68b50ddd187eb4351d" member_type: VOTER }
I20250623 01:57:42.581216 13237 leader_election.cc:304] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4a0ff7480744f68b50ddd187eb4351d; no voters:
I20250623 01:57:42.582296 13237 leader_election.cc:290] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:57:42.582629 13240 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:42.583791 13240 raft_consensus.cc:695] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 1 LEADER]: Becoming Leader. State: Replica: a4a0ff7480744f68b50ddd187eb4351d, State: Running, Role: LEADER
I20250623 01:57:42.584419 13240 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a0ff7480744f68b50ddd187eb4351d" member_type: VOTER }
I20250623 01:57:42.585008 13237 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:57:42.589867 13242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4a0ff7480744f68b50ddd187eb4351d. Latest consensus state: current_term: 1 leader_uuid: "a4a0ff7480744f68b50ddd187eb4351d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a0ff7480744f68b50ddd187eb4351d" member_type: VOTER } }
I20250623 01:57:42.589980 13241 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4a0ff7480744f68b50ddd187eb4351d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a0ff7480744f68b50ddd187eb4351d" member_type: VOTER } }
I20250623 01:57:42.590551 13242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:42.590580 13241 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:42.592630 13249 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:57:42.597051 13249 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:57:42.599459 12636 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250623 01:57:42.605034 13249 catalog_manager.cc:1349] Generated new cluster ID: 7337e4a3a5d54980b3b9ffa10d522b0f
I20250623 01:57:42.605293 13249 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:57:42.621326 13249 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:57:42.622466 13249 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:57:42.637219 13249 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d: Generated new TSK 0
I20250623 01:57:42.637773 13249 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:57:42.666339 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:42.671865 13258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:42.673024 13259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:42.674479 12636 server_base.cc:1048] running on GCE node
W20250623 01:57:42.674448 13261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:42.675597 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:57:42.675770 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:57:42.675884 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643862675874 us; error 0 us; skew 500 ppm
I20250623 01:57:42.676312 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:42.678481 12636 webserver.cc:469] Webserver started at http://127.12.87.1:40273/ using document root <none> and password file <none>
I20250623 01:57:42.678925 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:42.679075 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:42.679280 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:42.680275 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-0-root/instance:
uuid: "61244692ea674370a1678cee534ef92f"
format_stamp: "Formatted at 2025-06-23 01:57:42 on dist-test-slave-47g2"
I20250623 01:57:42.684343 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250623 01:57:42.687286 13266 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.687897 12636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250623 01:57:42.688170 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-0-root
uuid: "61244692ea674370a1678cee534ef92f"
format_stamp: "Formatted at 2025-06-23 01:57:42 on dist-test-slave-47g2"
I20250623 01:57:42.688431 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:42.715662 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:42.716861 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:42.718245 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:42.720340 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:42.720504 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.720736 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:42.720906 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.757417 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.1:44117
I20250623 01:57:42.757516 13328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.1:44117 every 8 connection(s)
I20250623 01:57:42.761924 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:42.768934 13333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:42.769805 13334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:42.772444 13336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:42.773636 12636 server_base.cc:1048] running on GCE node
I20250623 01:57:42.774071 13329 heartbeater.cc:344] Connected to a master server at 127.12.87.62:45311
I20250623 01:57:42.774428 13329 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:42.774971 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250623 01:57:42.775066 13329 heartbeater.cc:507] Master 127.12.87.62:45311 requested a full tablet report, sending...
W20250623 01:57:42.775195 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:57:42.775416 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643862775398 us; error 0 us; skew 500 ppm
I20250623 01:57:42.775902 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:42.777114 13202 ts_manager.cc:194] Registered new tserver with Master: 61244692ea674370a1678cee534ef92f (127.12.87.1:44117)
I20250623 01:57:42.778661 12636 webserver.cc:469] Webserver started at http://127.12.87.2:46061/ using document root <none> and password file <none>
I20250623 01:57:42.779289 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:42.779515 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:42.779523 13202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58026
I20250623 01:57:42.779994 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:42.781278 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-1-root/instance:
uuid: "05ca19107f7b442bb011af4486c39121"
format_stamp: "Formatted at 2025-06-23 01:57:42 on dist-test-slave-47g2"
I20250623 01:57:42.785362 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250623 01:57:42.788944 13341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.789839 12636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250623 01:57:42.790184 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-1-root
uuid: "05ca19107f7b442bb011af4486c39121"
format_stamp: "Formatted at 2025-06-23 01:57:42 on dist-test-slave-47g2"
I20250623 01:57:42.790503 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:42.803701 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:42.804761 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:42.806138 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:42.808255 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:42.808441 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.808667 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:42.808805 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.845587 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.2:33063
I20250623 01:57:42.845669 13403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.2:33063 every 8 connection(s)
I20250623 01:57:42.850121 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:42.856905 13407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:42.858817 13408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:42.861739 12636 server_base.cc:1048] running on GCE node
W20250623 01:57:42.862036 13410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:42.862560 13404 heartbeater.cc:344] Connected to a master server at 127.12.87.62:45311
I20250623 01:57:42.862892 13404 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:42.862944 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:57:42.863216 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:57:42.863396 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643862863376 us; error 0 us; skew 500 ppm
I20250623 01:57:42.863660 13404 heartbeater.cc:507] Master 127.12.87.62:45311 requested a full tablet report, sending...
I20250623 01:57:42.863951 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:42.865775 13202 ts_manager.cc:194] Registered new tserver with Master: 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:42.866629 12636 webserver.cc:469] Webserver started at http://127.12.87.3:41801/ using document root <none> and password file <none>
I20250623 01:57:42.867245 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:42.867468 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:42.867492 13202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58038
I20250623 01:57:42.867864 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:42.869072 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-2-root/instance:
uuid: "285843747942468fbf40a845620eae2a"
format_stamp: "Formatted at 2025-06-23 01:57:42 on dist-test-slave-47g2"
I20250623 01:57:42.873423 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250623 01:57:42.876317 13415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.876981 12636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250623 01:57:42.877234 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-2-root
uuid: "285843747942468fbf40a845620eae2a"
format_stamp: "Formatted at 2025-06-23 01:57:42 on dist-test-slave-47g2"
I20250623 01:57:42.877518 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750643830703842-12636-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:42.894409 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:42.895501 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:42.896814 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:42.898821 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:42.898996 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.899230 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:42.899369 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:42.936563 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.3:36095
I20250623 01:57:42.936655 13477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.3:36095 every 8 connection(s)
I20250623 01:57:42.952348 13478 heartbeater.cc:344] Connected to a master server at 127.12.87.62:45311
I20250623 01:57:42.952716 13478 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:42.953508 13478 heartbeater.cc:507] Master 127.12.87.62:45311 requested a full tablet report, sending...
I20250623 01:57:42.955150 13202 ts_manager.cc:194] Registered new tserver with Master: 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:42.956082 12636 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016637548s
I20250623 01:57:42.956568 13202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58044
I20250623 01:57:42.958894 12636 test_util.cc:276] Using random seed: -1986759917
I20250623 01:57:42.979259 13202 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58058:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 01:57:42.982064 13202 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 01:57:43.050416 13294 tablet_service.cc:1468] Processing CreateTablet for tablet 5de029fdba7b4b849e514cf0bb50e03c (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250623 01:57:43.051856 13294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de029fdba7b4b849e514cf0bb50e03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.052316 13293 tablet_service.cc:1468] Processing CreateTablet for tablet d0ba7025644e42a9919963b29cc750c7 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250623 01:57:43.052734 13292 tablet_service.cc:1468] Processing CreateTablet for tablet d51ca69a4cca450fb24989f8eeef180d (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250623 01:57:43.053575 13293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ba7025644e42a9919963b29cc750c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.054278 13292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51ca69a4cca450fb24989f8eeef180d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.059995 13291 tablet_service.cc:1468] Processing CreateTablet for tablet b6c53e621b464de98f002df9b8059689 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250623 01:57:43.062496 13290 tablet_service.cc:1468] Processing CreateTablet for tablet 13009913457c422c90edf7642617a9a6 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250623 01:57:43.063117 13285 tablet_service.cc:1468] Processing CreateTablet for tablet 14e2909a88a946f8b189545c6e424499 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250623 01:57:43.064513 13443 tablet_service.cc:1468] Processing CreateTablet for tablet 5de029fdba7b4b849e514cf0bb50e03c (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250623 01:57:43.064754 13289 tablet_service.cc:1468] Processing CreateTablet for tablet f6dadb6231a64e59890b64f5d1f9c378 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250623 01:57:43.065822 13443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de029fdba7b4b849e514cf0bb50e03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.067925 13287 tablet_service.cc:1468] Processing CreateTablet for tablet b2cd94319861432eaf6cfdbe97e2c970 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250623 01:57:43.070185 13286 tablet_service.cc:1468] Processing CreateTablet for tablet e41accfec71f47eb95bc728b0797841c (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250623 01:57:43.072774 13288 tablet_service.cc:1468] Processing CreateTablet for tablet 6760fbdc8bb940b09b98d4594d13a47f (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250623 01:57:43.073933 13291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c53e621b464de98f002df9b8059689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.074604 13286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41accfec71f47eb95bc728b0797841c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.077859 13285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e2909a88a946f8b189545c6e424499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.078586 13290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13009913457c422c90edf7642617a9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.079541 13287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2cd94319861432eaf6cfdbe97e2c970. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.080407 13442 tablet_service.cc:1468] Processing CreateTablet for tablet d0ba7025644e42a9919963b29cc750c7 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250623 01:57:43.080842 13289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6dadb6231a64e59890b64f5d1f9c378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.081744 13442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ba7025644e42a9919963b29cc750c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.093978 13440 tablet_service.cc:1468] Processing CreateTablet for tablet b6c53e621b464de98f002df9b8059689 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250623 01:57:43.095366 13440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c53e621b464de98f002df9b8059689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.091651 13441 tablet_service.cc:1468] Processing CreateTablet for tablet d51ca69a4cca450fb24989f8eeef180d (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250623 01:57:43.098551 13441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51ca69a4cca450fb24989f8eeef180d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.100371 13288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6760fbdc8bb940b09b98d4594d13a47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.102352 13439 tablet_service.cc:1468] Processing CreateTablet for tablet 13009913457c422c90edf7642617a9a6 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250623 01:57:43.103590 13439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13009913457c422c90edf7642617a9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.115754 13438 tablet_service.cc:1468] Processing CreateTablet for tablet f6dadb6231a64e59890b64f5d1f9c378 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250623 01:57:43.117687 13438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6dadb6231a64e59890b64f5d1f9c378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.128089 13492 tablet_bootstrap.cc:492] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.130188 13437 tablet_service.cc:1468] Processing CreateTablet for tablet 6760fbdc8bb940b09b98d4594d13a47f (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250623 01:57:43.131578 13437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6760fbdc8bb940b09b98d4594d13a47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.132433 13369 tablet_service.cc:1468] Processing CreateTablet for tablet 5de029fdba7b4b849e514cf0bb50e03c (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250623 01:57:43.132946 13364 tablet_service.cc:1468] Processing CreateTablet for tablet f6dadb6231a64e59890b64f5d1f9c378 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250623 01:57:43.133709 13369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de029fdba7b4b849e514cf0bb50e03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.134671 13492 tablet_bootstrap.cc:654] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.134490 13436 tablet_service.cc:1468] Processing CreateTablet for tablet b2cd94319861432eaf6cfdbe97e2c970 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250623 01:57:43.135133 13363 tablet_service.cc:1468] Processing CreateTablet for tablet 6760fbdc8bb940b09b98d4594d13a47f (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250623 01:57:43.135793 13436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2cd94319861432eaf6cfdbe97e2c970. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.136440 13363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6760fbdc8bb940b09b98d4594d13a47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.131211 13365 tablet_service.cc:1468] Processing CreateTablet for tablet 13009913457c422c90edf7642617a9a6 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250623 01:57:43.155802 13360 tablet_service.cc:1468] Processing CreateTablet for tablet 14e2909a88a946f8b189545c6e424499 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250623 01:57:43.167279 13361 tablet_service.cc:1468] Processing CreateTablet for tablet e41accfec71f47eb95bc728b0797841c (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250623 01:57:43.168635 13364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6dadb6231a64e59890b64f5d1f9c378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.173884 13365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13009913457c422c90edf7642617a9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.156302 13434 tablet_service.cc:1468] Processing CreateTablet for tablet 14e2909a88a946f8b189545c6e424499 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250623 01:57:43.191171 13362 tablet_service.cc:1468] Processing CreateTablet for tablet b2cd94319861432eaf6cfdbe97e2c970 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250623 01:57:43.192754 13360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e2909a88a946f8b189545c6e424499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.198020 13435 tablet_service.cc:1468] Processing CreateTablet for tablet e41accfec71f47eb95bc728b0797841c (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250623 01:57:43.199352 13435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41accfec71f47eb95bc728b0797841c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.209429 13362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2cd94319861432eaf6cfdbe97e2c970. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.209889 13493 tablet_bootstrap.cc:492] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.211411 13368 tablet_service.cc:1468] Processing CreateTablet for tablet d0ba7025644e42a9919963b29cc750c7 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250623 01:57:43.213982 13367 tablet_service.cc:1468] Processing CreateTablet for tablet d51ca69a4cca450fb24989f8eeef180d (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250623 01:57:43.218086 13366 tablet_service.cc:1468] Processing CreateTablet for tablet b6c53e621b464de98f002df9b8059689 (DEFAULT_TABLE table=test-workload [id=635da8c0a44444e48f35cd91b49ea7f0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250623 01:57:43.220536 13493 tablet_bootstrap.cc:654] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.226539 13361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41accfec71f47eb95bc728b0797841c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.227560 13366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c53e621b464de98f002df9b8059689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.231657 13367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51ca69a4cca450fb24989f8eeef180d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.237851 13368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ba7025644e42a9919963b29cc750c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.226712 13434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e2909a88a946f8b189545c6e424499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.248065 13492 tablet_bootstrap.cc:492] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.253731 13492 ts_tablet_manager.cc:1397] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.126s user 0.024s sys 0.055s
I20250623 01:57:43.256409 13492 raft_consensus.cc:357] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.258440 13492 raft_consensus.cc:383] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.263955 13496 tablet_bootstrap.cc:492] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.259374 13492 raft_consensus.cc:738] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.264524 13493 tablet_bootstrap.cc:492] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:43.267778 13493 ts_tablet_manager.cc:1397] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.002s
I20250623 01:57:43.268483 13492 consensus_queue.cc:260] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.278388 13493 raft_consensus.cc:357] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.279227 13493 raft_consensus.cc:383] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.279671 13493 raft_consensus.cc:738] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.280514 13493 consensus_queue.cc:260] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.284759 13329 heartbeater.cc:499] Master 127.12.87.62:45311 was elected leader, sending a full tablet report...
I20250623 01:57:43.287350 13492 ts_tablet_manager.cc:1428] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.033s user 0.005s sys 0.011s
I20250623 01:57:43.289170 13492 tablet_bootstrap.cc:492] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.297399 13492 tablet_bootstrap.cc:654] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.302137 13496 tablet_bootstrap.cc:654] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.307884 13478 heartbeater.cc:499] Master 127.12.87.62:45311 was elected leader, sending a full tablet report...
I20250623 01:57:43.313851 13492 tablet_bootstrap.cc:492] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.314368 13492 ts_tablet_manager.cc:1397] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.009s
I20250623 01:57:43.317142 13492 raft_consensus.cc:357] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.317965 13492 raft_consensus.cc:383] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.318315 13492 raft_consensus.cc:738] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.319058 13493 ts_tablet_manager.cc:1428] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.051s user 0.019s sys 0.011s
I20250623 01:57:43.319382 13496 tablet_bootstrap.cc:492] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.319164 13492 consensus_queue.cc:260] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.319797 13496 ts_tablet_manager.cc:1397] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.002s
I20250623 01:57:43.320351 13493 tablet_bootstrap.cc:492] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.323300 13492 ts_tablet_manager.cc:1428] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20250623 01:57:43.324213 13492 tablet_bootstrap.cc:492] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.324466 13496 raft_consensus.cc:357] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.325206 13496 raft_consensus.cc:383] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.325510 13496 raft_consensus.cc:738] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.326179 13496 consensus_queue.cc:260] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.327358 13493 tablet_bootstrap.cc:654] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.330572 13492 tablet_bootstrap.cc:654] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.342032 13492 tablet_bootstrap.cc:492] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.342523 13492 ts_tablet_manager.cc:1397] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20250623 01:57:43.345139 13492 raft_consensus.cc:357] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.345854 13492 raft_consensus.cc:383] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.346143 13492 raft_consensus.cc:738] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.346925 13492 consensus_queue.cc:260] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.350752 13492 ts_tablet_manager.cc:1428] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250623 01:57:43.351640 13492 tablet_bootstrap.cc:492] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.357580 13492 tablet_bootstrap.cc:654] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.367403 13404 heartbeater.cc:499] Master 127.12.87.62:45311 was elected leader, sending a full tablet report...
I20250623 01:57:43.368008 13496 ts_tablet_manager.cc:1428] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.048s user 0.009s sys 0.037s
I20250623 01:57:43.368451 13493 tablet_bootstrap.cc:492] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:43.368887 13493 ts_tablet_manager.cc:1397] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.030s
I20250623 01:57:43.368920 13496 tablet_bootstrap.cc:492] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.370832 13492 tablet_bootstrap.cc:492] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.371331 13492 ts_tablet_manager.cc:1397] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20250623 01:57:43.371574 13493 raft_consensus.cc:357] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.372313 13493 raft_consensus.cc:383] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.372620 13493 raft_consensus.cc:738] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.373507 13493 consensus_queue.cc:260] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.374805 13498 raft_consensus.cc:491] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.375136 13496 tablet_bootstrap.cc:654] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.375265 13498 raft_consensus.cc:513] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.379369 13492 raft_consensus.cc:357] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.380162 13492 raft_consensus.cc:383] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.380515 13492 raft_consensus.cc:738] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.381332 13492 consensus_queue.cc:260] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.382714 13493 ts_tablet_manager.cc:1428] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20250623 01:57:43.383597 13493 tablet_bootstrap.cc:492] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.383702 13492 ts_tablet_manager.cc:1428] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250623 01:57:43.384769 13492 tablet_bootstrap.cc:492] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.390779 13492 tablet_bootstrap.cc:654] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.392235 13496 tablet_bootstrap.cc:492] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.392683 13496 ts_tablet_manager.cc:1397] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.000s
I20250623 01:57:43.394723 13493 tablet_bootstrap.cc:654] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.395136 13496 raft_consensus.cc:357] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.395831 13496 raft_consensus.cc:383] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.397096 13498 leader_election.cc:290] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:43.402685 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c53e621b464de98f002df9b8059689" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f" is_pre_election: true
W20250623 01:57:43.404742 13419 leader_election.cc:343] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 61244692ea674370a1678cee534ef92f (127.12.87.1:44117): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.396129 13496 raft_consensus.cc:738] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.407953 13496 consensus_queue.cc:260] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.411242 13496 ts_tablet_manager.cc:1428] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250623 01:57:43.412396 13496 tablet_bootstrap.cc:492] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.420456 13496 tablet_bootstrap.cc:654] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.424878 13497 raft_consensus.cc:491] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.425390 13497 raft_consensus.cc:513] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.427536 13492 tablet_bootstrap.cc:492] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.427695 13497 leader_election.cc:290] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.428227 13492 ts_tablet_manager.cc:1397] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.029s
I20250623 01:57:43.440230 13492 raft_consensus.cc:357] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.440994 13493 tablet_bootstrap.cc:492] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:43.441203 13492 raft_consensus.cc:383] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.441555 13493 ts_tablet_manager.cc:1397] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.058s user 0.022s sys 0.027s
I20250623 01:57:43.441675 13492 raft_consensus.cc:738] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.442612 13492 consensus_queue.cc:260] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.444334 13493 raft_consensus.cc:357] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.445004 13493 raft_consensus.cc:383] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.445281 13493 raft_consensus.cc:738] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.446564 13492 ts_tablet_manager.cc:1428] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20250623 01:57:43.446898 13493 consensus_queue.cc:260] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.448101 13492 tablet_bootstrap.cc:492] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.449587 13493 ts_tablet_manager.cc:1428] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250623 01:57:43.449381 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c53e621b464de98f002df9b8059689" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.451238 13493 tablet_bootstrap.cc:492] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.451509 13379 raft_consensus.cc:2466] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 0.
I20250623 01:57:43.453409 13419 leader_election.cc:304] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters: 61244692ea674370a1678cee534ef92f
I20250623 01:57:43.454432 13498 raft_consensus.cc:2802] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.454762 13498 raft_consensus.cc:491] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.455082 13498 raft_consensus.cc:3058] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.455726 13492 tablet_bootstrap.cc:654] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.456264 13496 tablet_bootstrap.cc:492] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.456684 13496 ts_tablet_manager.cc:1397] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.003s
I20250623 01:57:43.457808 13493 tablet_bootstrap.cc:654] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.459187 13496 raft_consensus.cc:357] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.459862 13496 raft_consensus.cc:383] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.460126 13496 raft_consensus.cc:738] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.461491 13498 raft_consensus.cc:513] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.469521 13496 consensus_queue.cc:260] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.474457 13496 ts_tablet_manager.cc:1428] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s
I20250623 01:57:43.475490 13496 tablet_bootstrap.cc:492] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.478073 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c53e621b464de98f002df9b8059689" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f"
W20250623 01:57:43.479552 13419 leader_election.cc:343] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 61244692ea674370a1678cee534ef92f (127.12.87.1:44117): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.480738 13498 leader_election.cc:290] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:43.482375 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c53e621b464de98f002df9b8059689" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:43.482919 13379 raft_consensus.cc:3058] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.485556 13497 raft_consensus.cc:491] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.486012 13497 raft_consensus.cc:513] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.487941 13497 leader_election.cc:290] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.488521 13501 raft_consensus.cc:491] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.488963 13501 raft_consensus.cc:513] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.490078 13379 raft_consensus.cc:2466] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 1.
I20250623 01:57:43.491151 13501 leader_election.cc:290] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.491132 13419 leader_election.cc:304] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters: 61244692ea674370a1678cee534ef92f
I20250623 01:57:43.492334 13498 raft_consensus.cc:2802] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:43.493006 13498 raft_consensus.cc:695] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Becoming Leader. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:43.492884 13496 tablet_bootstrap.cc:654] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.493767 13498 consensus_queue.cc:237] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.496524 13505 raft_consensus.cc:491] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.496964 13505 raft_consensus.cc:513] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.499235 13505 leader_election.cc:290] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:43.501045 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f" is_pre_election: true
I20250623 01:57:43.501847 13304 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 0.
I20250623 01:57:43.503670 13419 leader_election.cc:304] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.504449 13505 raft_consensus.cc:2802] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.504792 13505 raft_consensus.cc:491] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.505131 13505 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.506361 13496 tablet_bootstrap.cc:492] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.506783 13496 ts_tablet_manager.cc:1397] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.005s
I20250623 01:57:43.509119 13496 raft_consensus.cc:357] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.509941 13496 raft_consensus.cc:383] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.510229 13496 raft_consensus.cc:738] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.510937 13496 consensus_queue.cc:260] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.511497 13505 raft_consensus.cc:513] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
W20250623 01:57:43.512193 13330 tablet.cc:2378] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:57:43.513278 13496 ts_tablet_manager.cc:1428] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250623 01:57:43.513854 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:43.514257 13496 tablet_bootstrap.cc:492] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.514400 13304 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.520618 13496 tablet_bootstrap.cc:654] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.521068 13304 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 1.
I20250623 01:57:43.522785 13419 leader_election.cc:304] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.532766 13496 tablet_bootstrap.cc:492] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.533252 13496 ts_tablet_manager.cc:1397] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20250623 01:57:43.535866 13517 raft_consensus.cc:2802] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:43.535898 13496 raft_consensus.cc:357] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.536731 13496 raft_consensus.cc:383] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.537003 13496 raft_consensus.cc:738] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.537115 13505 leader_election.cc:290] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:43.540817 13496 consensus_queue.cc:260] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.544104 13496 ts_tablet_manager.cc:1428] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250623 01:57:43.545104 13496 tablet_bootstrap.cc:492] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.550544 13517 raft_consensus.cc:695] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Becoming Leader. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:43.551075 13496 tablet_bootstrap.cc:654] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.551445 13517 consensus_queue.cc:237] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.555778 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:43.558216 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
W20250623 01:57:43.560359 13270 leader_election.cc:343] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 285843747942468fbf40a845620eae2a (127.12.87.3:36095): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.567665 13496 tablet_bootstrap.cc:492] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.568192 13496 ts_tablet_manager.cc:1397] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.007s
I20250623 01:57:43.571055 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.571527 13496 raft_consensus.cc:357] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.572207 13496 raft_consensus.cc:383] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.572386 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.572537 13496 raft_consensus.cc:738] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.572988 13378 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:43.573392 13496 consensus_queue.cc:260] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.574223 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.574313 13379 raft_consensus.cc:2466] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:43.574833 13378 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 0.
I20250623 01:57:43.576153 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:43.576655 13378 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.581291 13496 ts_tablet_manager.cc:1428] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20250623 01:57:43.582459 13496 tablet_bootstrap.cc:492] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.587365 13270 leader_election.cc:304] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters: 285843747942468fbf40a845620eae2a
I20250623 01:57:43.588510 13270 leader_election.cc:304] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters: 285843747942468fbf40a845620eae2a
I20250623 01:57:43.588857 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f" is_pre_election: true
I20250623 01:57:43.584997 13378 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 1.
I20250623 01:57:43.589413 13497 raft_consensus.cc:2762] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250623 01:57:43.590129 13497 raft_consensus.cc:2802] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.590458 13497 raft_consensus.cc:491] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.590847 13497 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.595561 13496 tablet_bootstrap.cc:654] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
W20250623 01:57:43.595463 13250 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:57:43.603621 13304 raft_consensus.cc:2391] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 05ca19107f7b442bb011af4486c39121 in current term 1: Already voted for candidate 285843747942468fbf40a845620eae2a in this term.
I20250623 01:57:43.600562 13497 raft_consensus.cc:513] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.606943 13497 leader_election.cc:290] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.607329 13496 tablet_bootstrap.cc:492] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.608984 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:43.614688 13496 ts_tablet_manager.cc:1397] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20250623 01:57:43.616568 13345 leader_election.cc:304] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05ca19107f7b442bb011af4486c39121; no voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f
I20250623 01:57:43.622896 13493 tablet_bootstrap.cc:492] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:43.623482 13493 ts_tablet_manager.cc:1397] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.172s user 0.059s sys 0.092s
I20250623 01:57:43.626999 13493 raft_consensus.cc:357] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.627882 13493 raft_consensus.cc:383] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.628279 13493 raft_consensus.cc:738] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.629210 13493 consensus_queue.cc:260] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.632508 13493 ts_tablet_manager.cc:1428] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20250623 01:57:43.618028 13501 raft_consensus.cc:2747] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:43.633641 13493 tablet_bootstrap.cc:492] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.634971 13501 raft_consensus.cc:491] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.635443 13501 raft_consensus.cc:513] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.635491 13201 catalog_manager.cc:5582] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 285843747942468fbf40a845620eae2a (127.12.87.3). New cstate: current_term: 1 leader_uuid: "285843747942468fbf40a845620eae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:43.637444 13501 leader_election.cc:290] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.639554 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f" is_pre_election: true
W20250623 01:57:43.641255 13345 leader_election.cc:343] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 61244692ea674370a1678cee534ef92f (127.12.87.1:44117): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.641769 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
W20250623 01:57:43.643083 13345 leader_election.cc:343] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 285843747942468fbf40a845620eae2a (127.12.87.3:36095): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.642983 13496 raft_consensus.cc:357] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.643508 13345 leader_election.cc:304] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05ca19107f7b442bb011af4486c39121; no voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f
I20250623 01:57:43.643682 13496 raft_consensus.cc:383] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.644001 13496 raft_consensus.cc:738] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.644675 13493 tablet_bootstrap.cc:654] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.644021 13497 raft_consensus.cc:491] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.645265 13497 raft_consensus.cc:513] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.647001 13492 tablet_bootstrap.cc:492] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.647158 13497 leader_election.cc:290] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.647498 13492 ts_tablet_manager.cc:1397] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.200s user 0.059s sys 0.095s
I20250623 01:57:43.647923 13501 raft_consensus.cc:2747] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:43.644701 13496 consensus_queue.cc:260] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.650111 13496 ts_tablet_manager.cc:1428] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.035s user 0.005s sys 0.002s
I20250623 01:57:43.651010 13496 tablet_bootstrap.cc:492] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.651036 13492 raft_consensus.cc:357] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.652180 13492 raft_consensus.cc:383] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 01:57:43.652603 13270 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:33063, user_credentials={real_user=slave}} blocked reactor thread for 64160.2us
I20250623 01:57:43.652599 13492 raft_consensus.cc:738] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.653820 13492 consensus_queue.cc:260] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.654634 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:43.656219 13492 ts_tablet_manager.cc:1428] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250623 01:57:43.656960 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de029fdba7b4b849e514cf0bb50e03c" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:43.657303 13496 tablet_bootstrap.cc:654] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.660569 13492 tablet_bootstrap.cc:492] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.657629 13453 raft_consensus.cc:2466] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:43.661674 13493 tablet_bootstrap.cc:492] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:43.662098 13493 ts_tablet_manager.cc:1397] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.002s
W20250623 01:57:43.658638 13270 leader_election.cc:343] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 285843747942468fbf40a845620eae2a (127.12.87.3:36095): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.664844 13493 raft_consensus.cc:357] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.665520 13270 leader_election.cc:304] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.665882 13493 raft_consensus.cc:383] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.666371 13493 raft_consensus.cc:738] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.666671 13497 raft_consensus.cc:2802] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.667073 13492 tablet_bootstrap.cc:654] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.667335 13497 raft_consensus.cc:491] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.667896 13497 raft_consensus.cc:3058] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.676179 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:43.676939 13378 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.680068 13492 tablet_bootstrap.cc:492] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.680697 13492 ts_tablet_manager.cc:1397] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20250623 01:57:43.684228 13492 raft_consensus.cc:357] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.685104 13492 raft_consensus.cc:383] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.685462 13492 raft_consensus.cc:738] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.667369 13493 consensus_queue.cc:260] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.686403 13492 consensus_queue.cc:260] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.687870 13378 raft_consensus.cc:2466] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:43.690344 13523 raft_consensus.cc:491] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.690846 13493 ts_tablet_manager.cc:1428] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20250623 01:57:43.690949 13523 raft_consensus.cc:513] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.692165 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de029fdba7b4b849e514cf0bb50e03c" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.693415 13523 leader_election.cc:290] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.695333 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:43.696703 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
W20250623 01:57:43.696985 13270 leader_election.cc:343] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 285843747942468fbf40a845620eae2a (127.12.87.3:36095): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.697489 13379 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:43.697831 13270 leader_election.cc:304] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters: 285843747942468fbf40a845620eae2a
I20250623 01:57:43.699149 13492 ts_tablet_manager.cc:1428] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.018s user 0.010s sys 0.001s
I20250623 01:57:43.700414 13492 tablet_bootstrap.cc:492] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.699590 13493 tablet_bootstrap.cc:492] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.707557 13523 raft_consensus.cc:2802] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election won for term 1
W20250623 01:57:43.715703 13270 leader_election.cc:343] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.716715 13270 leader_election.cc:304] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters: 285843747942468fbf40a845620eae2a
I20250623 01:57:43.718036 13496 tablet_bootstrap.cc:492] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.718626 13496 ts_tablet_manager.cc:1397] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.037s
I20250623 01:57:43.723088 13497 raft_consensus.cc:513] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.724961 13497 leader_election.cc:290] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.725343 13492 tablet_bootstrap.cc:654] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.726270 13532 raft_consensus.cc:491] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.731048 13532 raft_consensus.cc:513] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.733189 13501 raft_consensus.cc:491] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.734014 13493 tablet_bootstrap.cc:654] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.733780 13501 raft_consensus.cc:513] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.736141 13501 leader_election.cc:290] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.726642 13533 raft_consensus.cc:2802] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.726528 13496 raft_consensus.cc:357] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.736898 13496 raft_consensus.cc:383] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.737280 13496 raft_consensus.cc:738] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.738011 13496 consensus_queue.cc:260] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.738452 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de029fdba7b4b849e514cf0bb50e03c" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:43.738973 13379 raft_consensus.cc:3058] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.738868 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41accfec71f47eb95bc728b0797841c" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f" is_pre_election: true
I20250623 01:57:43.739758 13304 raft_consensus.cc:2466] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 0.
I20250623 01:57:43.740060 13496 ts_tablet_manager.cc:1428] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.021s user 0.004s sys 0.003s
I20250623 01:57:43.740978 13496 tablet_bootstrap.cc:492] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.740995 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41accfec71f47eb95bc728b0797841c" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:43.743197 13345 leader_election.cc:304] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.743847 13493 tablet_bootstrap.cc:492] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
W20250623 01:57:43.744017 13345 leader_election.cc:343] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 285843747942468fbf40a845620eae2a (127.12.87.3:36095): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.744402 13493 ts_tablet_manager.cc:1397] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.045s user 0.014s sys 0.002s
I20250623 01:57:43.745641 13379 raft_consensus.cc:2466] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:43.746452 13496 tablet_bootstrap.cc:654] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.746822 13501 raft_consensus.cc:2802] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.747193 13501 raft_consensus.cc:491] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.747538 13501 raft_consensus.cc:3058] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.747606 13493 raft_consensus.cc:357] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.748443 13533 raft_consensus.cc:491] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.748457 13493 raft_consensus.cc:383] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.748989 13533 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.749055 13493 raft_consensus.cc:738] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.750077 13493 consensus_queue.cc:260] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.752207 13493 ts_tablet_manager.cc:1428] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250623 01:57:43.753062 13493 tablet_bootstrap.cc:492] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.752755 13270 leader_election.cc:304] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.753659 13497 raft_consensus.cc:491] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.754343 13534 raft_consensus.cc:2802] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:43.754233 13497 raft_consensus.cc:513] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.755867 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de029fdba7b4b849e514cf0bb50e03c" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:43.756405 13453 raft_consensus.cc:3058] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.756623 13497 leader_election.cc:290] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.759006 13452 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ba7025644e42a9919963b29cc750c7" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:43.759474 13532 leader_election.cc:290] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.759685 13452 raft_consensus.cc:2466] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:43.761229 13270 leader_election.cc:304] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.761513 13452 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41accfec71f47eb95bc728b0797841c" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:43.762336 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41accfec71f47eb95bc728b0797841c" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.746263 13201 catalog_manager.cc:5582] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 285843747942468fbf40a845620eae2a (127.12.87.3). New cstate: current_term: 1 leader_uuid: "285843747942468fbf40a845620eae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:43.764660 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ba7025644e42a9919963b29cc750c7" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.765760 13532 raft_consensus.cc:2802] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.770274 13532 raft_consensus.cc:491] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.770756 13532 raft_consensus.cc:3058] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:57:43.766433 13270 leader_election.cc:343] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 285843747942468fbf40a845620eae2a (127.12.87.3:36095): Illegal state: must be running to vote when last-logged opid is not known
W20250623 01:57:43.777992 13270 leader_election.cc:343] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.780987 13493 tablet_bootstrap.cc:654] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.783448 13496 tablet_bootstrap.cc:492] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:43.784475 13523 raft_consensus.cc:695] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:43.781373 13501 raft_consensus.cc:513] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.785097 13496 ts_tablet_manager.cc:1397] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.027s
I20250623 01:57:43.785658 13378 raft_consensus.cc:2391] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61244692ea674370a1678cee534ef92f in current term 1: Already voted for candidate 05ca19107f7b442bb011af4486c39121 in this term.
I20250623 01:57:43.785635 13523 consensus_queue.cc:237] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.787854 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41accfec71f47eb95bc728b0797841c" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:43.787689 13496 raft_consensus.cc:357] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.788460 13496 raft_consensus.cc:383] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.788445 13304 raft_consensus.cc:3058] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.790462 13452 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41accfec71f47eb95bc728b0797841c" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a"
W20250623 01:57:43.791671 13345 leader_election.cc:343] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 285843747942468fbf40a845620eae2a (127.12.87.3:36095): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.777222 13453 raft_consensus.cc:2466] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:43.792901 13501 leader_election.cc:290] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.792586 13270 leader_election.cc:304] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61244692ea674370a1678cee534ef92f; no voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a
I20250623 01:57:43.783515 13533 raft_consensus.cc:513] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.795864 13304 raft_consensus.cc:2466] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 1.
I20250623 01:57:43.796233 13496 raft_consensus.cc:738] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.796749 13492 tablet_bootstrap.cc:492] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.796904 13534 raft_consensus.cc:695] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:43.799897 13345 leader_election.cc:304] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters: 285843747942468fbf40a845620eae2a
I20250623 01:57:43.799779 13496 consensus_queue.cc:260] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.800350 13493 tablet_bootstrap.cc:492] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:43.800889 13493 ts_tablet_manager.cc:1397] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.003s
I20250623 01:57:43.800276 13534 consensus_queue.cc:237] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.801210 13497 raft_consensus.cc:2747] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:43.801271 13501 raft_consensus.cc:2802] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:43.804183 13493 raft_consensus.cc:357] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.805012 13493 raft_consensus.cc:383] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.805375 13493 raft_consensus.cc:738] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.805475 13492 ts_tablet_manager.cc:1397] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.105s user 0.019s sys 0.000s
I20250623 01:57:43.806150 13493 consensus_queue.cc:260] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.810467 13533 leader_election.cc:290] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.813153 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:43.808692 13492 raft_consensus.cc:357] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.813745 13378 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.814047 13492 raft_consensus.cc:383] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.814448 13492 raft_consensus.cc:738] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.815016 13202 catalog_manager.cc:5582] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 0 to 1, leader changed from <none> to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 1 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:43.815224 13492 consensus_queue.cc:260] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.819729 13532 raft_consensus.cc:513] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.820842 13378 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:43.822991 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:43.823575 13453 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.824131 13493 ts_tablet_manager.cc:1428] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250623 01:57:43.824236 13532 leader_election.cc:290] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.823189 13492 ts_tablet_manager.cc:1428] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.017s user 0.005s sys 0.002s
I20250623 01:57:43.825480 13492 tablet_bootstrap.cc:492] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.825726 13493 tablet_bootstrap.cc:492] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.830888 13453 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:43.832338 13493 tablet_bootstrap.cc:654] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.832509 13492 tablet_bootstrap.cc:654] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.836469 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ba7025644e42a9919963b29cc750c7" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:43.838629 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ba7025644e42a9919963b29cc750c7" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:43.839293 13453 raft_consensus.cc:3058] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.845274 13492 tablet_bootstrap.cc:492] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.845921 13492 ts_tablet_manager.cc:1397] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.006s
I20250623 01:57:43.845914 13270 leader_election.cc:304] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.849987 13532 raft_consensus.cc:2802] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:43.850670 13532 raft_consensus.cc:695] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
W20250623 01:57:43.850734 13270 leader_election.cc:343] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:57:43.851863 13532 consensus_queue.cc:237] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.856632 13453 raft_consensus.cc:2466] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:43.859638 13270 leader_election.cc:304] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters: 05ca19107f7b442bb011af4486c39121
I20250623 01:57:43.860925 13534 raft_consensus.cc:2802] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:43.861704 13534 raft_consensus.cc:695] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:43.862809 13534 consensus_queue.cc:237] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.860812 13492 raft_consensus.cc:357] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.868300 13492 raft_consensus.cc:383] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.868700 13492 raft_consensus.cc:738] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.869693 13492 consensus_queue.cc:260] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.873384 13201 catalog_manager.cc:5582] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 0 to 1, leader changed from <none> to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 1 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:43.875118 13492 ts_tablet_manager.cc:1428] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20250623 01:57:43.876601 13492 tablet_bootstrap.cc:492] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f: Bootstrap starting.
I20250623 01:57:43.883687 13492 tablet_bootstrap.cc:654] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.884025 13493 tablet_bootstrap.cc:492] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:43.884477 13493 ts_tablet_manager.cc:1397] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.059s user 0.027s sys 0.020s
I20250623 01:57:43.887343 13493 raft_consensus.cc:357] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.888104 13304 raft_consensus.cc:3058] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.888214 13493 raft_consensus.cc:383] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.888659 13493 raft_consensus.cc:738] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.896407 13493 consensus_queue.cc:260] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.897768 13498 consensus_queue.cc:1035] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:57:43.898752 13496 ts_tablet_manager.cc:1428] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.113s user 0.032s sys 0.053s
I20250623 01:57:43.904320 13493 ts_tablet_manager.cc:1428] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20250623 01:57:43.905542 13496 tablet_bootstrap.cc:492] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121: Bootstrap starting.
I20250623 01:57:43.905535 13493 tablet_bootstrap.cc:492] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.911717 13501 raft_consensus.cc:695] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 1 LEADER]: Becoming Leader. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:57:43.912688 13496 tablet_bootstrap.cc:654] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.912621 13501 consensus_queue.cc:237] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.913645 13493 tablet_bootstrap.cc:654] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.925992 13493 tablet_bootstrap.cc:492] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:43.926600 13493 ts_tablet_manager.cc:1397] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20250623 01:57:43.929792 13493 raft_consensus.cc:357] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.930651 13493 raft_consensus.cc:383] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.930977 13493 raft_consensus.cc:738] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.931932 13493 consensus_queue.cc:260] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.939025 13493 ts_tablet_manager.cc:1428] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250623 01:57:43.941397 13493 tablet_bootstrap.cc:492] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a: Bootstrap starting.
I20250623 01:57:43.944025 13202 catalog_manager.cc:5582] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 0 to 1, leader changed from <none> to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 1 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:43.946595 13202 catalog_manager.cc:5582] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 0 to 1, leader changed from <none> to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 1 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:43.950779 13517 raft_consensus.cc:491] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.951403 13517 raft_consensus.cc:513] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.955915 13492 tablet_bootstrap.cc:492] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f: No bootstrap required, opened a new log
I20250623 01:57:43.956176 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f" is_pre_election: true
I20250623 01:57:43.956459 13492 ts_tablet_manager.cc:1397] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f: Time spent bootstrapping tablet: real 0.080s user 0.041s sys 0.029s
I20250623 01:57:43.956821 13505 raft_consensus.cc:491] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.957283 13505 raft_consensus.cc:513] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.957898 13532 raft_consensus.cc:491] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.958323 13532 raft_consensus.cc:513] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.959154 13505 leader_election.cc:290] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:43.959352 13492 raft_consensus.cc:357] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.960309 13532 leader_election.cc:290] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.961098 13492 raft_consensus.cc:383] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.961459 13492 raft_consensus.cc:738] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.962146 13492 consensus_queue.cc:260] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.962435 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.963068 13378 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:43.963068 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:43.964397 13453 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:43.965744 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f" is_pre_election: true
I20250623 01:57:43.966718 13517 leader_election.cc:290] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:43.967113 13270 leader_election.cc:304] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.967486 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.968103 13493 tablet_bootstrap.cc:654] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.968065 13378 raft_consensus.cc:2466] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 0.
I20250623 01:57:43.969105 13304 raft_consensus.cc:2466] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 0.
I20250623 01:57:43.969720 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:43.970703 13379 raft_consensus.cc:2391] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 285843747942468fbf40a845620eae2a in current term 1: Already voted for candidate 61244692ea674370a1678cee534ef92f in this term.
I20250623 01:57:43.971618 13532 raft_consensus.cc:2802] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.972069 13532 raft_consensus.cc:491] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.972590 13532 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.973445 13419 leader_election.cc:304] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.975284 13419 leader_election.cc:304] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 285843747942468fbf40a845620eae2a; no voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f
I20250623 01:57:43.976284 13517 raft_consensus.cc:2802] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.976641 13517 raft_consensus.cc:491] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.976979 13517 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.977121 13505 consensus_queue.cc:1035] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:43.978505 13505 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.978438 13200 catalog_manager.cc:5582] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05ca19107f7b442bb011af4486c39121 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "05ca19107f7b442bb011af4486c39121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:43.982296 13532 raft_consensus.cc:513] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.983247 13492 ts_tablet_manager.cc:1428] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20250623 01:57:43.985383 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:43.989332 13378 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.989464 13532 leader_election.cc:290] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:43.990067 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:43.990500 13453 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.994992 13453 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:43.985967 13517 raft_consensus.cc:513] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:43.996068 13270 leader_election.cc:304] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:43.996302 13378 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:43.997011 13532 raft_consensus.cc:2802] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:43.998389 13532 raft_consensus.cc:695] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:43.999364 13532 consensus_queue.cc:237] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.000978 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:44.001277 13493 tablet_bootstrap.cc:492] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a: No bootstrap required, opened a new log
I20250623 01:57:44.001821 13493 ts_tablet_manager.cc:1397] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.008s
I20250623 01:57:43.997368 13517 leader_election.cc:290] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:44.001744 13304 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:44.001230 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:44.003270 13378 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:44.004374 13534 raft_consensus.cc:491] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:44.004690 13493 raft_consensus.cc:357] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.004997 13534 raft_consensus.cc:513] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.005616 13493 raft_consensus.cc:383] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:44.006027 13493 raft_consensus.cc:738] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Initialized, Role: FOLLOWER
I20250623 01:57:44.006821 13493 consensus_queue.cc:260] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.007184 13534 leader_election.cc:290] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:44.009363 13493 ts_tablet_manager.cc:1428] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250623 01:57:43.985298 13505 raft_consensus.cc:2747] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:44.010980 13304 raft_consensus.cc:2466] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 1.
I20250623 01:57:44.014087 13419 leader_election.cc:304] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:44.015151 13505 raft_consensus.cc:2802] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:44.015955 13505 raft_consensus.cc:695] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Becoming Leader. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:44.017040 13505 consensus_queue.cc:237] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.018185 13378 raft_consensus.cc:2466] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 1.
I20250623 01:57:44.011543 13496 tablet_bootstrap.cc:492] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121: No bootstrap required, opened a new log
I20250623 01:57:44.011559 13377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e2909a88a946f8b189545c6e424499" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:44.018962 13377 raft_consensus.cc:2466] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:44.019004 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e2909a88a946f8b189545c6e424499" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a" is_pre_election: true
I20250623 01:57:44.019657 13453 raft_consensus.cc:2466] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 0.
I20250623 01:57:44.019878 13496 ts_tablet_manager.cc:1397] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121: Time spent bootstrapping tablet: real 0.118s user 0.014s sys 0.010s
I20250623 01:57:44.020149 13270 leader_election.cc:304] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:44.020884 13534 raft_consensus.cc:2802] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:44.021215 13534 raft_consensus.cc:491] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:44.021577 13534 raft_consensus.cc:3058] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:44.028261 13534 raft_consensus.cc:513] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.030193 13534 leader_election.cc:290] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:44.032254 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e2909a88a946f8b189545c6e424499" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:44.032301 13496 raft_consensus.cc:357] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.032939 13453 raft_consensus.cc:3058] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:44.033206 13496 raft_consensus.cc:383] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:44.033389 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e2909a88a946f8b189545c6e424499" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:44.033614 13496 raft_consensus.cc:738] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Initialized, Role: FOLLOWER
I20250623 01:57:44.033990 13379 raft_consensus.cc:3058] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:44.034333 13496 consensus_queue.cc:260] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.037398 13496 ts_tablet_manager.cc:1428] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20250623 01:57:44.041843 13453 raft_consensus.cc:2466] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:44.042057 13379 raft_consensus.cc:2466] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 1.
I20250623 01:57:44.043397 13270 leader_election.cc:304] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:44.044222 13534 raft_consensus.cc:2802] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:44.044854 13534 raft_consensus.cc:695] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:44.048169 13534 consensus_queue.cc:237] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.051318 13200 catalog_manager.cc:5582] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 285843747942468fbf40a845620eae2a (127.12.87.3). New cstate: current_term: 1 leader_uuid: "285843747942468fbf40a845620eae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:44.053645 13199 catalog_manager.cc:5582] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 0 to 1, leader changed from <none> to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 1 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:44.067026 13199 catalog_manager.cc:5582] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 0 to 1, leader changed from <none> to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 1 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:44.075402 13505 consensus_queue.cc:1035] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.084955 13505 consensus_queue.cc:1035] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:57:44.174489 13453 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b6c53e621b464de98f002df9b8059689"
dest_uuid: "285843747942468fbf40a845620eae2a"
mode: GRACEFUL
new_leader_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:32842
I20250623 01:57:44.175004 13453 raft_consensus.cc:604] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Received request to transfer leadership to TS 61244692ea674370a1678cee534ef92f
I20250623 01:57:44.179577 13453 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "13009913457c422c90edf7642617a9a6"
dest_uuid: "285843747942468fbf40a845620eae2a"
mode: GRACEFUL
new_leader_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:32842
I20250623 01:57:44.180117 13453 raft_consensus.cc:604] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Received request to transfer leadership to TS 61244692ea674370a1678cee534ef92f
I20250623 01:57:44.183187 13505 raft_consensus.cc:491] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:44.183559 13505 raft_consensus.cc:513] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.185398 13505 leader_election.cc:290] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:44.186820 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41accfec71f47eb95bc728b0797841c" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61244692ea674370a1678cee534ef92f" is_pre_election: true
I20250623 01:57:44.187091 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41accfec71f47eb95bc728b0797841c" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca19107f7b442bb011af4486c39121" is_pre_election: true
I20250623 01:57:44.187515 13304 raft_consensus.cc:2391] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 285843747942468fbf40a845620eae2a in current term 1: Already voted for candidate 05ca19107f7b442bb011af4486c39121 in this term.
I20250623 01:57:44.188560 13419 leader_election.cc:304] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 285843747942468fbf40a845620eae2a; no voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f
I20250623 01:57:44.189200 13505 raft_consensus.cc:3058] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:44.193439 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:37498
I20250623 01:57:44.193962 13304 raft_consensus.cc:604] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Received request to transfer leadership to TS 05ca19107f7b442bb011af4486c39121
I20250623 01:57:44.195178 13505 raft_consensus.cc:2747] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:57:44.196710 13534 consensus_queue.cc:1035] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.198871 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:37498
I20250623 01:57:44.199357 13304 raft_consensus.cc:604] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Received request to transfer leadership to TS 05ca19107f7b442bb011af4486c39121
I20250623 01:57:44.205605 13453 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970"
dest_uuid: "285843747942468fbf40a845620eae2a"
mode: GRACEFUL
new_leader_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:32842
I20250623 01:57:44.206207 13453 raft_consensus.cc:604] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Received request to transfer leadership to TS 05ca19107f7b442bb011af4486c39121
I20250623 01:57:44.208331 13523 consensus_queue.cc:1035] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.216406 13534 raft_consensus.cc:991] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f: : Instructing follower 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:57:44.216816 13534 raft_consensus.cc:1079] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Signalling peer 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:57:44.218480 13379 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f"
dest_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:57304
I20250623 01:57:44.219003 13379 raft_consensus.cc:491] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:44.219367 13379 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.222394 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "14e2909a88a946f8b189545c6e424499"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:37498
I20250623 01:57:44.222841 13304 raft_consensus.cc:604] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Received request to transfer leadership to TS 285843747942468fbf40a845620eae2a
I20250623 01:57:44.225275 13379 raft_consensus.cc:513] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.226852 13379 leader_election.cc:290] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 2 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:44.227617 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:44.227870 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:44.228132 13304 raft_consensus.cc:3053] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:44.228430 13453 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.228474 13304 raft_consensus.cc:738] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:44.229354 13304 consensus_queue.cc:260] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.230619 13304 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.233258 13453 raft_consensus.cc:2466] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 2.
I20250623 01:57:44.234126 13345 leader_election.cc:304] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:57:44.234695 13501 raft_consensus.cc:2802] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:44.235054 13501 raft_consensus.cc:695] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Becoming Leader. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:57:44.235738 13304 raft_consensus.cc:2466] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 2.
I20250623 01:57:44.235729 13501 consensus_queue.cc:237] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.241669 13199 catalog_manager.cc:5582] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 reported cstate change: term changed from 1 to 2, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 05ca19107f7b442bb011af4486c39121 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "05ca19107f7b442bb011af4486c39121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:44.251008 13534 consensus_queue.cc:1035] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.259821 13523 consensus_queue.cc:1035] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.278191 13379 raft_consensus.cc:3058] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:44.283792 13532 consensus_queue.cc:1035] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.293809 13533 consensus_queue.cc:1035] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.389003 13501 consensus_queue.cc:1035] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.389943 13533 consensus_queue.cc:1035] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.395864 13501 consensus_queue.cc:1035] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.399408 13497 raft_consensus.cc:991] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f: : Instructing follower 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:57:44.399847 13497 raft_consensus.cc:1079] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Signalling peer 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:57:44.401611 13377 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378"
dest_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:57304
I20250623 01:57:44.402175 13377 raft_consensus.cc:491] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:44.402552 13377 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.406533 13534 consensus_queue.cc:1035] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.409641 13377 raft_consensus.cc:513] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.412400 13303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:44.412976 13303 raft_consensus.cc:3053] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:44.413277 13303 raft_consensus.cc:738] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:44.414077 13303 consensus_queue.cc:260] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.414995 13303 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.418762 13379 raft_consensus.cc:1238] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Rejecting Update request from peer 61244692ea674370a1678cee534ef92f for earlier term 1. Current term is 2. Ops: []
I20250623 01:57:44.419978 13377 leader_election.cc:290] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 2 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:44.423198 13303 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 2.
I20250623 01:57:44.424284 13345 leader_election.cc:304] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:44.425328 13542 raft_consensus.cc:2802] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:44.425814 13542 raft_consensus.cc:695] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Becoming Leader. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:57:44.426226 13523 consensus_queue.cc:1035] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.426638 13542 consensus_queue.cc:237] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.432366 13520 consensus_queue.cc:1035] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:57:44.436129 13453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:44.436897 13453 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.438535 13520 raft_consensus.cc:991] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a: : Instructing follower 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:44.440002 13498 raft_consensus.cc:1079] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Signalling peer 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:44.445299 13523 consensus_queue.cc:1035] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.447204 13505 raft_consensus.cc:991] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a: : Instructing follower 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:57:44.447609 13505 raft_consensus.cc:1079] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Signalling peer 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:57:44.449187 13379 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970"
dest_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:57298
I20250623 01:57:44.451413 13379 raft_consensus.cc:491] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:44.451808 13379 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.453125 13303 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b6c53e621b464de98f002df9b8059689"
dest_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:37468
I20250623 01:57:44.453888 13303 raft_consensus.cc:491] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:44.454262 13303 raft_consensus.cc:3058] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.457048 13534 raft_consensus.cc:991] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f: : Instructing follower 285843747942468fbf40a845620eae2a to start an election
I20250623 01:57:44.457564 13534 raft_consensus.cc:1079] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Signalling peer 285843747942468fbf40a845620eae2a to start an election
I20250623 01:57:44.460031 13498 consensus_queue.cc:1035] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.461643 13451 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "14e2909a88a946f8b189545c6e424499"
dest_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:32834
I20250623 01:57:44.461742 13303 raft_consensus.cc:513] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.461472 13200 catalog_manager.cc:5582] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 reported cstate change: term changed from 1 to 2, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 05ca19107f7b442bb011af4486c39121 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "05ca19107f7b442bb011af4486c39121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:44.462287 13451 raft_consensus.cc:491] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:44.462841 13451 raft_consensus.cc:3058] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.447427 13453 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 2.
I20250623 01:57:44.463814 13303 leader_election.cc:290] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 2 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:44.465940 13378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c53e621b464de98f002df9b8059689" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:44.473630 13378 raft_consensus.cc:3058] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.475909 13451 raft_consensus.cc:513] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.466998 13449 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6c53e621b464de98f002df9b8059689" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:44.477140 13450 raft_consensus.cc:1238] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Rejecting Update request from peer 61244692ea674370a1678cee534ef92f for earlier term 1. Current term is 2. Ops: []
I20250623 01:57:44.477382 13449 raft_consensus.cc:3053] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:44.477748 13449 raft_consensus.cc:738] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:44.478516 13449 consensus_queue.cc:260] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.479713 13449 raft_consensus.cc:3058] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.481879 13497 consensus_queue.cc:1035] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.481799 13451 leader_election.cc:290] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 2 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:44.486164 13303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e2909a88a946f8b189545c6e424499" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:44.487013 13376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e2909a88a946f8b189545c6e424499" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:44.487322 13449 raft_consensus.cc:2466] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 2.
I20250623 01:57:44.487656 13376 raft_consensus.cc:3058] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.467821 13523 consensus_queue.cc:1035] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.458370 13379 raft_consensus.cc:513] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.489513 13303 raft_consensus.cc:3053] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:44.489326 13533 consensus_queue.cc:1046] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.489832 13303 raft_consensus.cc:738] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:44.491025 13303 consensus_queue.cc:260] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.492655 13303 raft_consensus.cc:3058] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.494589 13378 raft_consensus.cc:2466] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 2.
I20250623 01:57:44.497445 13376 raft_consensus.cc:2466] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 2.
I20250623 01:57:44.498952 13419 leader_election.cc:304] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:57:44.499333 13303 raft_consensus.cc:2466] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 2.
W20250623 01:57:44.499843 13497 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250623 01:57:44.500054 13505 raft_consensus.cc:2802] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:44.500730 13505 raft_consensus.cc:695] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 2 LEADER]: Becoming Leader. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:44.488404 13270 leader_election.cc:304] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:44.501859 13505 consensus_queue.cc:237] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.502714 13523 raft_consensus.cc:2802] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:44.503482 13523 raft_consensus.cc:695] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 2 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:44.504345 13523 consensus_queue.cc:237] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.505769 13379 leader_election.cc:290] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 2 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:44.507377 13451 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:44.508143 13451 raft_consensus.cc:3053] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:44.508571 13451 raft_consensus.cc:738] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:44.510633 13451 consensus_queue.cc:260] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.511593 13451 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.517659 13303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:44.518229 13303 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.519196 13451 raft_consensus.cc:2466] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 2.
I20250623 01:57:44.520324 13345 leader_election.cc:304] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:57:44.521173 13542 raft_consensus.cc:2802] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:44.521672 13542 raft_consensus.cc:695] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Becoming Leader. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:57:44.522446 13542 consensus_queue.cc:237] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.524905 13202 catalog_manager.cc:5582] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a reported cstate change: term changed from 1 to 2, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 285843747942468fbf40a845620eae2a (127.12.87.3). New cstate: current_term: 2 leader_uuid: "285843747942468fbf40a845620eae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:44.527715 13303 raft_consensus.cc:2466] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 2.
I20250623 01:57:44.533876 13199 catalog_manager.cc:5582] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 reported cstate change: term changed from 1 to 2, leader changed from 285843747942468fbf40a845620eae2a (127.12.87.3) to 05ca19107f7b442bb011af4486c39121 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "05ca19107f7b442bb011af4486c39121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:44.541965 13200 catalog_manager.cc:5582] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 1 to 2, leader changed from 285843747942468fbf40a845620eae2a (127.12.87.3) to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 2 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:44.683368 13520 raft_consensus.cc:991] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a: : Instructing follower 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:44.683779 13520 raft_consensus.cc:1079] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Signalling peer 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:44.685254 13303 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "13009913457c422c90edf7642617a9a6"
dest_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:37468
I20250623 01:57:44.685811 13303 raft_consensus.cc:491] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:44.686261 13303 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.691051 13303 raft_consensus.cc:513] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.692555 13303 leader_election.cc:290] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 2 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:44.693724 13451 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:44.694360 13451 raft_consensus.cc:3053] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:44.694674 13451 raft_consensus.cc:738] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:44.694715 13376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:44.695410 13376 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.695503 13451 consensus_queue.cc:260] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.696712 13451 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:44.702262 13376 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 2.
I20250623 01:57:44.703238 13451 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 2.
I20250623 01:57:44.703317 13270 leader_election.cc:304] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:44.704092 13533 raft_consensus.cc:2802] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:44.704493 13533 raft_consensus.cc:695] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 2 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:44.705152 13533 consensus_queue.cc:237] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:44.711346 13200 catalog_manager.cc:5582] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 1 to 2, leader changed from 285843747942468fbf40a845620eae2a (127.12.87.3) to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 2 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:44.737287 13451 raft_consensus.cc:1273] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:44.738320 13542 consensus_queue.cc:1035] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:44.744845 13303 raft_consensus.cc:1273] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:44.746670 13541 consensus_queue.cc:1035] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:44.924721 13451 raft_consensus.cc:1273] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:44.925742 13541 consensus_queue.cc:1035] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:44.932760 13303 raft_consensus.cc:1273] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:44.934520 13541 consensus_queue.cc:1035] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:44.977663 13451 raft_consensus.cc:1273] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:44.978649 13541 consensus_queue.cc:1035] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:44.986606 13303 raft_consensus.cc:1273] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:44.987824 13541 consensus_queue.cc:1035] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:44.989845 13303 raft_consensus.cc:1273] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:44.994012 13520 consensus_queue.cc:1035] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:44.994724 13451 raft_consensus.cc:1273] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:44.996333 13533 consensus_queue.cc:1035] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:44.999617 13377 raft_consensus.cc:1273] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:45.000231 13376 raft_consensus.cc:1273] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:45.006124 13497 consensus_queue.cc:1035] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:45.006678 13505 consensus_queue.cc:1035] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:45.165490 13451 raft_consensus.cc:1273] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:45.166654 13533 consensus_queue.cc:1035] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:45.174352 13377 raft_consensus.cc:1273] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:45.176302 13497 consensus_queue.cc:1035] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:47.250682 12636 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20250623 01:57:47.251260 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:47.266574 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "13009913457c422c90edf7642617a9a6"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:37500
I20250623 01:57:47.267015 13304 raft_consensus.cc:604] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 2 LEADER]: Received request to transfer leadership to TS 285843747942468fbf40a845620eae2a
I20250623 01:57:47.269019 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:37500
I20250623 01:57:47.269443 13304 raft_consensus.cc:604] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Received request to transfer leadership to TS 285843747942468fbf40a845620eae2a
I20250623 01:57:47.270253 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250623 01:57:47.270632 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:47.555059 13590 raft_consensus.cc:991] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f: : Instructing follower 285843747942468fbf40a845620eae2a to start an election
I20250623 01:57:47.555495 13587 raft_consensus.cc:1079] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 2 LEADER]: Signalling peer 285843747942468fbf40a845620eae2a to start an election
I20250623 01:57:47.556870 13451 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "13009913457c422c90edf7642617a9a6"
dest_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:32834
I20250623 01:57:47.557300 13451 raft_consensus.cc:491] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:47.557574 13451 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:47.562078 13451 raft_consensus.cc:513] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:47.563339 13451 leader_election.cc:290] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 3 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:47.564164 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:47.564359 13377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:47.564735 13304 raft_consensus.cc:3053] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:47.564821 13377 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:47.565121 13304 raft_consensus.cc:738] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:47.565905 13304 consensus_queue.cc:260] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:47.567076 13304 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:47.571185 13377 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 3.
I20250623 01:57:47.572351 13419 leader_election.cc:304] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:57:47.573099 13505 raft_consensus.cc:2802] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:47.573626 13304 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 3.
I20250623 01:57:47.574115 13505 raft_consensus.cc:695] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 3 LEADER]: Becoming Leader. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:47.574856 13505 consensus_queue.cc:237] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:47.581563 13202 catalog_manager.cc:5582] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a reported cstate change: term changed from 2 to 3, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 285843747942468fbf40a845620eae2a (127.12.87.3). New cstate: current_term: 3 leader_uuid: "285843747942468fbf40a845620eae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:47.660714 13587 raft_consensus.cc:991] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f: : Instructing follower 285843747942468fbf40a845620eae2a to start an election
I20250623 01:57:47.661254 13590 raft_consensus.cc:1079] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Signalling peer 285843747942468fbf40a845620eae2a to start an election
I20250623 01:57:47.662644 13451 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d"
dest_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:32834
I20250623 01:57:47.663070 13451 raft_consensus.cc:491] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:47.663333 13451 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:47.667737 13451 raft_consensus.cc:513] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:47.669137 13451 leader_election.cc:290] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 2 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:57:47.669879 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:57:47.670351 13304 raft_consensus.cc:3053] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:57:47.670286 13377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:47.670655 13304 raft_consensus.cc:738] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:47.670859 13377 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:47.671362 13304 consensus_queue.cc:260] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:47.672214 13304 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:47.675209 13377 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 2.
I20250623 01:57:47.676075 13419 leader_election.cc:304] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:57:47.676707 13304 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 2.
I20250623 01:57:47.676849 13505 raft_consensus.cc:2802] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:47.677295 13505 raft_consensus.cc:695] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 2 LEADER]: Becoming Leader. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:47.678056 13505 consensus_queue.cc:237] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:47.684357 13202 catalog_manager.cc:5582] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a reported cstate change: term changed from 1 to 2, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 285843747942468fbf40a845620eae2a (127.12.87.3). New cstate: current_term: 2 leader_uuid: "285843747942468fbf40a845620eae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:47.983191 13377 raft_consensus.cc:1273] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:47.984617 13505 consensus_queue.cc:1035] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:47.993505 13304 raft_consensus.cc:1273] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:47.994668 13505 consensus_queue.cc:1035] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:48.246915 13377 raft_consensus.cc:1273] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:48.248155 13505 consensus_queue.cc:1035] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:48.256319 13304 raft_consensus.cc:1273] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:48.258706 13604 consensus_queue.cc:1035] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:57:48.271329 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:48.276821 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:48.277241 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:49.278127 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:49.283341 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:49.283746 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:50.284593 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:50.290287 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:50.290635 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:51.291587 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:51.297868 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:51.298252 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:52.299089 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:52.304512 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:52.304919 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:53.305728 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:53.310672 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:53.311017 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:54.311810 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:54.316614 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:54.316979 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:55.317755 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:55.322753 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:55.323103 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:57:56.445650 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:57:56.450479 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:57:56.450865 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250623 01:57:56.761011 13325 debug-util.cc:398] Leaking SignalData structure 0x7b08001d4cc0 after lost signal to thread 12639
W20250623 01:57:56.761785 13325 debug-util.cc:398] Leaking SignalData structure 0x7b08001ce780 after lost signal to thread 13236
W20250623 01:57:56.762563 13325 debug-util.cc:398] Leaking SignalData structure 0x7b080015f100 after lost signal to thread 13328
W20250623 01:57:56.763270 13325 debug-util.cc:398] Leaking SignalData structure 0x7b08000e8840 after lost signal to thread 13403
W20250623 01:57:56.763947 13325 debug-util.cc:398] Leaking SignalData structure 0x7b080012a980 after lost signal to thread 13477
I20250623 01:57:57.483095 12636 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20250623 01:57:57.493359 13449 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d"
dest_uuid: "285843747942468fbf40a845620eae2a"
mode: GRACEFUL
new_leader_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:32842
I20250623 01:57:57.494287 13449 raft_consensus.cc:604] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 2 LEADER]: Received request to transfer leadership to TS 61244692ea674370a1678cee534ef92f
I20250623 01:57:57.501771 13449 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "13009913457c422c90edf7642617a9a6"
dest_uuid: "285843747942468fbf40a845620eae2a"
mode: GRACEFUL
new_leader_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:32842
I20250623 01:57:57.502338 13449 raft_consensus.cc:604] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 3 LEADER]: Received request to transfer leadership to TS 61244692ea674370a1678cee534ef92f
I20250623 01:57:57.523502 13377 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378"
dest_uuid: "05ca19107f7b442bb011af4486c39121"
mode: GRACEFUL
new_leader_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:40254
I20250623 01:57:57.523972 13377 raft_consensus.cc:604] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Received request to transfer leadership to TS 61244692ea674370a1678cee534ef92f
I20250623 01:57:57.528761 13377 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f"
dest_uuid: "05ca19107f7b442bb011af4486c39121"
mode: GRACEFUL
new_leader_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:40254
I20250623 01:57:57.529237 13377 raft_consensus.cc:604] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Received request to transfer leadership to TS 61244692ea674370a1678cee534ef92f
I20250623 01:57:57.533721 13377 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970"
dest_uuid: "05ca19107f7b442bb011af4486c39121"
mode: GRACEFUL
new_leader_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:40254
I20250623 01:57:57.534082 13377 raft_consensus.cc:604] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Received request to transfer leadership to TS 61244692ea674370a1678cee534ef92f
I20250623 01:57:57.685659 13617 raft_consensus.cc:991] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121: : Instructing follower 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:57.686105 13617 raft_consensus.cc:1079] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Signalling peer 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:57.687445 13304 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f"
dest_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:37484
I20250623 01:57:57.687942 13304 raft_consensus.cc:491] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:57.688340 13304 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:57.695995 13304 raft_consensus.cc:513] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.698065 13304 leader_election.cc:290] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 3 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:57.698805 13377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:57.699241 13450 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:57.699590 13377 raft_consensus.cc:3053] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:57.699769 13450 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:57.699963 13377 raft_consensus.cc:738] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:57:57.700770 13377 consensus_queue.cc:260] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.702189 13377 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:57.706915 13450 raft_consensus.cc:2466] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 3.
I20250623 01:57:57.708271 13270 leader_election.cc:304] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:57.708729 13377 raft_consensus.cc:2466] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 3.
I20250623 01:57:57.709160 13587 raft_consensus.cc:2802] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:57.710141 13587 raft_consensus.cc:695] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:57.711035 13587 consensus_queue.cc:237] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.723093 13199 catalog_manager.cc:5582] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 2 to 3, leader changed from 05ca19107f7b442bb011af4486c39121 (127.12.87.2) to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 3 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:57.818179 13624 raft_consensus.cc:991] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a: : Instructing follower 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:57.818802 13612 raft_consensus.cc:1079] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 3 LEADER]: Signalling peer 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:57.820386 13304 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "13009913457c422c90edf7642617a9a6"
dest_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:37468
I20250623 01:57:57.820919 13304 raft_consensus.cc:491] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:57.821261 13304 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:57.827987 13304 raft_consensus.cc:513] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.829730 13304 leader_election.cc:290] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 4 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:57.830335 13377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:57.830693 13450 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13009913457c422c90edf7642617a9a6" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:57.830869 13377 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:57.831287 13450 raft_consensus.cc:3053] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:57:57.831622 13450 raft_consensus.cc:738] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:57.832442 13450 consensus_queue.cc:260] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.833772 13450 raft_consensus.cc:3058] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:57.836102 13377 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 4.
I20250623 01:57:57.837491 13270 leader_election.cc:304] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:57.840662 13450 raft_consensus.cc:2466] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 4.
I20250623 01:57:57.841044 13587 raft_consensus.cc:2802] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:57:57.841564 13587 raft_consensus.cc:695] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 4 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:57.842267 13587 consensus_queue.cc:237] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.849694 13199 catalog_manager.cc:5582] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 3 to 4, leader changed from 285843747942468fbf40a845620eae2a (127.12.87.3) to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 4 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:57.959429 13624 raft_consensus.cc:991] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a: : Instructing follower 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:57.959954 13612 raft_consensus.cc:1079] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 2 LEADER]: Signalling peer 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:57.961470 13304 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d"
dest_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:37468
I20250623 01:57:57.962004 13304 raft_consensus.cc:491] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:57.962320 13304 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:57.968720 13304 raft_consensus.cc:513] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.970439 13304 leader_election.cc:290] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 3 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:57.971133 13377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:57.971638 13377 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:57.971796 13450 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:57.972381 13450 raft_consensus.cc:3053] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:57.972718 13450 raft_consensus.cc:738] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:57:57.973516 13450 consensus_queue.cc:260] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.974829 13450 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:57.975713 13377 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 3.
I20250623 01:57:57.976462 13270 leader_election.cc:304] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:57.977146 13587 raft_consensus.cc:2802] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:57.977492 13587 raft_consensus.cc:695] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:57.978235 13587 consensus_queue.cc:237] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:57.981858 13450 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 3.
I20250623 01:57:57.986052 13199 catalog_manager.cc:5582] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 2 to 3, leader changed from 285843747942468fbf40a845620eae2a (127.12.87.3) to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 3 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:58.006222 13611 raft_consensus.cc:991] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121: : Instructing follower 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:58.006671 13617 raft_consensus.cc:1079] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Signalling peer 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:58.008260 13304 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378"
dest_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:37484
I20250623 01:57:58.008792 13304 raft_consensus.cc:491] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:58.009133 13304 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:58.015386 13304 raft_consensus.cc:513] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:58.017695 13377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:58.018268 13377 raft_consensus.cc:3053] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:58.017855 13304 leader_election.cc:290] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 3 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:58.018612 13377 raft_consensus.cc:738] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:57:58.018469 13450 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:58.019533 13377 consensus_queue.cc:260] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:58.020272 13450 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:58.021100 13377 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:58.027830 13450 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 3.
I20250623 01:57:58.028579 13377 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 3.
I20250623 01:57:58.028873 13270 leader_election.cc:304] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:58.029698 13587 raft_consensus.cc:2802] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:58.030160 13587 raft_consensus.cc:695] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:58.030864 13587 consensus_queue.cc:237] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:58.038946 13199 catalog_manager.cc:5582] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 2 to 3, leader changed from 05ca19107f7b442bb011af4486c39121 (127.12.87.2) to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 3 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:58.077395 13611 raft_consensus.cc:991] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121: : Instructing follower 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:58.077904 13617 raft_consensus.cc:1079] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Signalling peer 61244692ea674370a1678cee534ef92f to start an election
I20250623 01:57:58.079247 13304 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970"
dest_uuid: "61244692ea674370a1678cee534ef92f"
from {username='slave'} at 127.0.0.1:37484
I20250623 01:57:58.079634 13304 raft_consensus.cc:491] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:58.079917 13304 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:58.083758 13304 raft_consensus.cc:513] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:58.084900 13304 leader_election.cc:290] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 3 election: Requested vote from peers 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:57:58.085628 13377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:57:58.086127 13377 raft_consensus.cc:3053] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:58.086062 13450 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2cd94319861432eaf6cfdbe97e2c970" candidate_uuid: "61244692ea674370a1678cee534ef92f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:57:58.086465 13377 raft_consensus.cc:738] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:57:58.086617 13450 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:58.087213 13377 consensus_queue.cc:260] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:58.088344 13377 raft_consensus.cc:3058] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:58.092895 13450 raft_consensus.cc:2466] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 3.
I20250623 01:57:58.094254 13377 raft_consensus.cc:2466] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61244692ea674370a1678cee534ef92f in term 3.
I20250623 01:57:58.095201 13270 leader_election.cc:304] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285843747942468fbf40a845620eae2a, 61244692ea674370a1678cee534ef92f; no voters:
I20250623 01:57:58.095737 13587 raft_consensus.cc:2802] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:57:58.096125 13587 raft_consensus.cc:695] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Becoming Leader. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:57:58.096818 13587 consensus_queue.cc:237] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:57:58.104815 13199 catalog_manager.cc:5582] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f reported cstate change: term changed from 2 to 3, leader changed from 05ca19107f7b442bb011af4486c39121 (127.12.87.2) to 61244692ea674370a1678cee534ef92f (127.12.87.1). New cstate: current_term: 3 leader_uuid: "61244692ea674370a1678cee534ef92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:57:58.180747 13377 raft_consensus.cc:1273] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:58.181967 13587 consensus_queue.cc:1035] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:58.197288 13450 raft_consensus.cc:1273] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:58.202569 13632 consensus_queue.cc:1035] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:57:58.431694 13450 raft_consensus.cc:1273] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:58.432902 13632 consensus_queue.cc:1035] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:58.435791 13449 raft_consensus.cc:1273] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:58.437126 13632 consensus_queue.cc:1035] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:58.441004 13449 raft_consensus.cc:1273] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:58.441592 13377 raft_consensus.cc:1273] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:58.444104 13632 consensus_queue.cc:1035] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:57:58.445695 13632 consensus_queue.cc:1035] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:58.456887 13376 raft_consensus.cc:1273] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:58.460345 13377 raft_consensus.cc:1273] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:58.468065 13632 consensus_queue.cc:1035] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:57:58.481194 13638 consensus_queue.cc:1035] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250623 01:57:58.500849 13450 raft_consensus.cc:1273] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:58.505522 13642 consensus_queue.cc:1035] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:57:58.530074 13377 raft_consensus.cc:1273] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Refusing update from remote peer 61244692ea674370a1678cee534ef92f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:57:58.531466 13641 consensus_queue.cc:1035] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:00.567243 12636 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20250623 01:58:00.567804 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:00.574219 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5de029fdba7b4b849e514cf0bb50e03c"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:37500
I20250623 01:58:00.574610 13304 raft_consensus.cc:604] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Received request to transfer leadership to TS 05ca19107f7b442bb011af4486c39121
I20250623 01:58:00.576608 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:37500
I20250623 01:58:00.576960 13304 raft_consensus.cc:604] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Received request to transfer leadership to TS 05ca19107f7b442bb011af4486c39121
I20250623 01:58:00.578606 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:37500
I20250623 01:58:00.578998 13304 raft_consensus.cc:604] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Received request to transfer leadership to TS 285843747942468fbf40a845620eae2a
I20250623 01:58:00.580672 13304 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378"
dest_uuid: "61244692ea674370a1678cee534ef92f"
mode: GRACEFUL
new_leader_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:37500
I20250623 01:58:00.580991 13304 raft_consensus.cc:604] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Received request to transfer leadership to TS 285843747942468fbf40a845620eae2a
I20250623 01:58:00.581743 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250623 01:58:00.582115 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:00.863834 13648 raft_consensus.cc:991] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f: : Instructing follower 285843747942468fbf40a845620eae2a to start an election
I20250623 01:58:00.864279 13647 raft_consensus.cc:1079] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Signalling peer 285843747942468fbf40a845620eae2a to start an election
I20250623 01:58:00.865445 13449 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378"
dest_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:32834
I20250623 01:58:00.865813 13449 raft_consensus.cc:491] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:58:00.866030 13449 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:00.870424 13449 raft_consensus.cc:513] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:00.872505 13449 leader_election.cc:290] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 4 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:58:00.873208 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:58:00.873550 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6dadb6231a64e59890b64f5d1f9c378" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:58:00.873661 13304 raft_consensus.cc:3053] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:00.874009 13304 raft_consensus.cc:738] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:58:00.874011 13379 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:00.874612 13304 consensus_queue.cc:260] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:00.875384 13304 raft_consensus.cc:3058] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:00.878508 13379 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 4.
I20250623 01:58:00.879205 13304 raft_consensus.cc:2466] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 4.
I20250623 01:58:00.879619 13419 leader_election.cc:304] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:58:00.880291 13612 raft_consensus.cc:2802] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:00.881021 13612 raft_consensus.cc:695] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 4 LEADER]: Becoming Leader. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:58:00.881642 13612 consensus_queue.cc:237] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:00.887684 13200 catalog_manager.cc:5582] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a reported cstate change: term changed from 3 to 4, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 285843747942468fbf40a845620eae2a (127.12.87.3). New cstate: current_term: 4 leader_uuid: "285843747942468fbf40a845620eae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:00.912791 13648 raft_consensus.cc:991] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f: : Instructing follower 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:58:00.913179 13647 raft_consensus.cc:1079] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Signalling peer 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:58:00.914351 13379 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f"
dest_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:57304
I20250623 01:58:00.914747 13379 raft_consensus.cc:491] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:58:00.914989 13379 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:00.919000 13379 raft_consensus.cc:513] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:00.920473 13379 leader_election.cc:290] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 4 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:58:00.921229 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:58:00.921425 13449 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6760fbdc8bb940b09b98d4594d13a47f" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:58:00.921720 13304 raft_consensus.cc:3053] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:00.921939 13449 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:00.922139 13304 raft_consensus.cc:738] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:58:00.922894 13304 consensus_queue.cc:260] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:00.923868 13304 raft_consensus.cc:3058] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:00.926506 13449 raft_consensus.cc:2466] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 4.
I20250623 01:58:00.927356 13345 leader_election.cc:304] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:58:00.927873 13617 raft_consensus.cc:2802] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:00.928704 13304 raft_consensus.cc:2466] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 4.
I20250623 01:58:00.929014 13617 raft_consensus.cc:695] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 4 LEADER]: Becoming Leader. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:58:00.929838 13617 consensus_queue.cc:237] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:00.935307 13200 catalog_manager.cc:5582] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 reported cstate change: term changed from 3 to 4, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 05ca19107f7b442bb011af4486c39121 (127.12.87.2). New cstate: current_term: 4 leader_uuid: "05ca19107f7b442bb011af4486c39121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:01.022011 13648 raft_consensus.cc:991] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f: : Instructing follower 285843747942468fbf40a845620eae2a to start an election
I20250623 01:58:01.022408 13647 raft_consensus.cc:1079] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Signalling peer 285843747942468fbf40a845620eae2a to start an election
I20250623 01:58:01.023533 13449 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d"
dest_uuid: "285843747942468fbf40a845620eae2a"
from {username='slave'} at 127.0.0.1:32834
I20250623 01:58:01.023928 13449 raft_consensus.cc:491] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:58:01.024127 13449 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:01.027918 13449 raft_consensus.cc:513] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:01.029515 13449 leader_election.cc:290] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 4 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063)
I20250623 01:58:01.030154 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:58:01.030354 13379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51ca69a4cca450fb24989f8eeef180d" candidate_uuid: "285843747942468fbf40a845620eae2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "05ca19107f7b442bb011af4486c39121"
I20250623 01:58:01.030655 13304 raft_consensus.cc:3053] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:01.030858 13379 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:01.030997 13304 raft_consensus.cc:738] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:58:01.031693 13304 consensus_queue.cc:260] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:01.032380 13304 raft_consensus.cc:3058] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:01.034981 13379 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 4.
I20250623 01:58:01.035988 13419 leader_election.cc:304] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:58:01.036305 13304 raft_consensus.cc:2466] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285843747942468fbf40a845620eae2a in term 4.
I20250623 01:58:01.036721 13612 raft_consensus.cc:2802] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:01.037199 13612 raft_consensus.cc:695] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 4 LEADER]: Becoming Leader. State: Replica: 285843747942468fbf40a845620eae2a, State: Running, Role: LEADER
I20250623 01:58:01.038026 13612 consensus_queue.cc:237] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:01.061892 13200 catalog_manager.cc:5582] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a reported cstate change: term changed from 3 to 4, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 285843747942468fbf40a845620eae2a (127.12.87.3). New cstate: current_term: 4 leader_uuid: "285843747942468fbf40a845620eae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:01.069861 13648 raft_consensus.cc:991] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f: : Instructing follower 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:58:01.070868 13647 raft_consensus.cc:1079] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Signalling peer 05ca19107f7b442bb011af4486c39121 to start an election
I20250623 01:58:01.072937 13379 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5de029fdba7b4b849e514cf0bb50e03c"
dest_uuid: "05ca19107f7b442bb011af4486c39121"
from {username='slave'} at 127.0.0.1:57304
I20250623 01:58:01.073691 13379 raft_consensus.cc:491] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:58:01.074205 13379 raft_consensus.cc:3058] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:01.081996 13379 raft_consensus.cc:513] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:01.083257 13379 leader_election.cc:290] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 2 election: Requested vote from peers 61244692ea674370a1678cee534ef92f (127.12.87.1:44117), 285843747942468fbf40a845620eae2a (127.12.87.3:36095)
I20250623 01:58:01.084206 13304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de029fdba7b4b849e514cf0bb50e03c" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "61244692ea674370a1678cee534ef92f"
I20250623 01:58:01.084666 13449 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de029fdba7b4b849e514cf0bb50e03c" candidate_uuid: "05ca19107f7b442bb011af4486c39121" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "285843747942468fbf40a845620eae2a"
I20250623 01:58:01.084743 13304 raft_consensus.cc:3053] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:01.085095 13304 raft_consensus.cc:738] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61244692ea674370a1678cee534ef92f, State: Running, Role: LEADER
I20250623 01:58:01.085152 13449 raft_consensus.cc:3058] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:01.085752 13304 consensus_queue.cc:260] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:01.086840 13304 raft_consensus.cc:3058] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:01.089991 13449 raft_consensus.cc:2466] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 2.
I20250623 01:58:01.090929 13345 leader_election.cc:304] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca19107f7b442bb011af4486c39121, 285843747942468fbf40a845620eae2a; no voters:
I20250623 01:58:01.091792 13617 raft_consensus.cc:2802] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:01.091970 13304 raft_consensus.cc:2466] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca19107f7b442bb011af4486c39121 in term 2.
I20250623 01:58:01.092176 13617 raft_consensus.cc:695] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Becoming Leader. State: Replica: 05ca19107f7b442bb011af4486c39121, State: Running, Role: LEADER
I20250623 01:58:01.093181 13617 consensus_queue.cc:237] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } }
I20250623 01:58:01.100688 13200 catalog_manager.cc:5582] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 reported cstate change: term changed from 1 to 2, leader changed from 61244692ea674370a1678cee534ef92f (127.12.87.1) to 05ca19107f7b442bb011af4486c39121 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "05ca19107f7b442bb011af4486c39121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:01.446147 13304 raft_consensus.cc:1273] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:58:01.447220 13612 consensus_queue.cc:1035] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:01.457677 13379 raft_consensus.cc:1273] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:58:01.458750 13612 consensus_queue.cc:1035] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:01.525578 13449 raft_consensus.cc:1273] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:58:01.526522 13617 consensus_queue.cc:1035] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:01.535964 13304 raft_consensus.cc:1273] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:58:01.536140 13450 raft_consensus.cc:1273] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:01.538170 13617 consensus_queue.cc:1035] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:01.538640 13666 consensus_queue.cc:1035] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285843747942468fbf40a845620eae2a" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:01.551867 13304 raft_consensus.cc:1273] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Refusing update from remote peer 05ca19107f7b442bb011af4486c39121: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:01.553483 13617 consensus_queue.cc:1035] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:01.565064 13304 raft_consensus.cc:1273] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:58:01.566035 13612 consensus_queue.cc:1035] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61244692ea674370a1678cee534ef92f" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:01.570235 13379 raft_consensus.cc:1273] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Refusing update from remote peer 285843747942468fbf40a845620eae2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:58:01.572256 13649 consensus_queue.cc:1035] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca19107f7b442bb011af4486c39121" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:01.583195 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:01.590153 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:01.590495 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:02.591323 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:02.597828 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:02.598155 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:03.598845 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:03.603886 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:03.604207 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:04.605021 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:04.609944 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:04.610345 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:05.611116 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:05.616083 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:05.616436 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:06.617180 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:06.622231 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:06.622584 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:07.623538 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:07.628582 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:07.628966 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:08.629894 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:08.635675 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:08.636149 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:09.637015 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:09.641893 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:58:09.642241 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:10.668457 12636 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20250623 01:58:10.668895 12636 tablet_server.cc:178] TabletServer@127.12.87.1:0 shutting down...
I20250623 01:58:10.693791 12636 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250623 01:58:10.694420 12636 tablet_replica.cc:331] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.694993 12636 raft_consensus.cc:2241] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 2 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.695751 12636 raft_consensus.cc:2270] T b6c53e621b464de98f002df9b8059689 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.697582 12636 tablet_replica.cc:331] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.698100 12636 raft_consensus.cc:2241] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.698480 12636 raft_consensus.cc:2270] T d51ca69a4cca450fb24989f8eeef180d P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.699981 12636 tablet_replica.cc:331] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.700459 12636 raft_consensus.cc:2241] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.700850 12636 raft_consensus.cc:2270] T 14e2909a88a946f8b189545c6e424499 P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.702425 12636 tablet_replica.cc:331] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.702946 12636 raft_consensus.cc:2241] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 3 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.703624 12636 raft_consensus.cc:2270] T b2cd94319861432eaf6cfdbe97e2c970 P 61244692ea674370a1678cee534ef92f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.705106 12636 tablet_replica.cc:331] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.705577 12636 raft_consensus.cc:2241] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.705962 12636 raft_consensus.cc:2270] T 6760fbdc8bb940b09b98d4594d13a47f P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.707377 12636 tablet_replica.cc:331] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.707808 12636 raft_consensus.cc:2241] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.708146 12636 raft_consensus.cc:2270] T f6dadb6231a64e59890b64f5d1f9c378 P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.709637 12636 tablet_replica.cc:331] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.710067 12636 raft_consensus.cc:2241] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.710389 12636 raft_consensus.cc:2270] T e41accfec71f47eb95bc728b0797841c P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.712070 12636 tablet_replica.cc:331] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.712646 12636 raft_consensus.cc:2241] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.713023 12636 raft_consensus.cc:2270] T 5de029fdba7b4b849e514cf0bb50e03c P 61244692ea674370a1678cee534ef92f [term 2 FOLLOWER]: Raft consensus is shut down!
W20250623 01:58:10.713599 13419 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
I20250623 01:58:10.714754 12636 tablet_replica.cc:331] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.715291 12636 raft_consensus.cc:2241] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 4 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.715934 12636 raft_consensus.cc:2270] T 13009913457c422c90edf7642617a9a6 P 61244692ea674370a1678cee534ef92f [term 4 FOLLOWER]: Raft consensus is shut down!
W20250623 01:58:10.716502 13419 consensus_peers.cc:487] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a -> Peer 61244692ea674370a1678cee534ef92f (127.12.87.1:44117): Couldn't send request to peer 61244692ea674370a1678cee534ef92f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:10.717677 12636 tablet_replica.cc:331] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f: stopping tablet replica
I20250623 01:58:10.718123 12636 raft_consensus.cc:2241] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 1 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.718770 12636 raft_consensus.cc:2270] T d0ba7025644e42a9919963b29cc750c7 P 61244692ea674370a1678cee534ef92f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.763620 12636 tablet_server.cc:195] TabletServer@127.12.87.1:0 shutdown complete.
I20250623 01:58:10.782644 12636 tablet_server.cc:178] TabletServer@127.12.87.2:0 shutting down...
W20250623 01:58:10.788221 13419 consensus_peers.cc:487] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a -> Peer 05ca19107f7b442bb011af4486c39121 (127.12.87.2:33063): Couldn't send request to peer 05ca19107f7b442bb011af4486c39121. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:58:10.802162 13345 consensus_peers.cc:487] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 -> Peer 61244692ea674370a1678cee534ef92f (127.12.87.1:44117): Couldn't send request to peer 61244692ea674370a1678cee534ef92f. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:44117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:10.811288 12636 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250623 01:58:10.811936 12636 tablet_replica.cc:331] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.812928 12636 raft_consensus.cc:2241] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.813522 12636 raft_consensus.cc:2270] T d0ba7025644e42a9919963b29cc750c7 P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.815200 12636 tablet_replica.cc:331] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.815702 12636 raft_consensus.cc:2241] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.816042 12636 raft_consensus.cc:2270] T 14e2909a88a946f8b189545c6e424499 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.817631 12636 tablet_replica.cc:331] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.818058 12636 raft_consensus.cc:2241] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 2 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.818642 12636 raft_consensus.cc:2270] T 5de029fdba7b4b849e514cf0bb50e03c P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.820079 12636 tablet_replica.cc:331] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.820518 12636 raft_consensus.cc:2241] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 1 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.821161 12636 raft_consensus.cc:2270] T e41accfec71f47eb95bc728b0797841c P 05ca19107f7b442bb011af4486c39121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.822782 12636 tablet_replica.cc:331] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.823235 12636 raft_consensus.cc:2241] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 4 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.823808 12636 raft_consensus.cc:2270] T 6760fbdc8bb940b09b98d4594d13a47f P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.825280 12636 tablet_replica.cc:331] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.825795 12636 raft_consensus.cc:2241] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.826126 12636 raft_consensus.cc:2270] T f6dadb6231a64e59890b64f5d1f9c378 P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.827621 12636 tablet_replica.cc:331] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.828090 12636 raft_consensus.cc:2241] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.828439 12636 raft_consensus.cc:2270] T b2cd94319861432eaf6cfdbe97e2c970 P 05ca19107f7b442bb011af4486c39121 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.830037 12636 tablet_replica.cc:331] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.830471 12636 raft_consensus.cc:2241] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.830839 12636 raft_consensus.cc:2270] T d51ca69a4cca450fb24989f8eeef180d P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.832360 12636 tablet_replica.cc:331] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.832778 12636 raft_consensus.cc:2241] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.833117 12636 raft_consensus.cc:2270] T b6c53e621b464de98f002df9b8059689 P 05ca19107f7b442bb011af4486c39121 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.834746 12636 tablet_replica.cc:331] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121: stopping tablet replica
I20250623 01:58:10.835140 12636 raft_consensus.cc:2241] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.835536 12636 raft_consensus.cc:2270] T 13009913457c422c90edf7642617a9a6 P 05ca19107f7b442bb011af4486c39121 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.862323 12636 tablet_server.cc:195] TabletServer@127.12.87.2:0 shutdown complete.
I20250623 01:58:10.879812 12636 tablet_server.cc:178] TabletServer@127.12.87.3:0 shutting down...
I20250623 01:58:10.903695 12636 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250623 01:58:10.904353 12636 tablet_replica.cc:331] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.905270 12636 raft_consensus.cc:2241] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 2 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.906275 12636 raft_consensus.cc:2270] T 14e2909a88a946f8b189545c6e424499 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.907961 12636 tablet_replica.cc:331] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.908383 12636 raft_consensus.cc:2241] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.908752 12636 raft_consensus.cc:2270] T b2cd94319861432eaf6cfdbe97e2c970 P 285843747942468fbf40a845620eae2a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.910322 12636 tablet_replica.cc:331] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.910760 12636 raft_consensus.cc:2241] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 4 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.911437 12636 raft_consensus.cc:2270] T f6dadb6231a64e59890b64f5d1f9c378 P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.912972 12636 tablet_replica.cc:331] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.913465 12636 raft_consensus.cc:2241] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.913815 12636 raft_consensus.cc:2270] T 6760fbdc8bb940b09b98d4594d13a47f P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.915212 12636 tablet_replica.cc:331] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.915670 12636 raft_consensus.cc:2241] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.916014 12636 raft_consensus.cc:2270] T e41accfec71f47eb95bc728b0797841c P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.917474 12636 tablet_replica.cc:331] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.917914 12636 raft_consensus.cc:2241] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.918246 12636 raft_consensus.cc:2270] T 5de029fdba7b4b849e514cf0bb50e03c P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.919633 12636 tablet_replica.cc:331] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.920082 12636 raft_consensus.cc:2241] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.920441 12636 raft_consensus.cc:2270] T d0ba7025644e42a9919963b29cc750c7 P 285843747942468fbf40a845620eae2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.921969 12636 tablet_replica.cc:331] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.922401 12636 raft_consensus.cc:2241] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.922752 12636 raft_consensus.cc:2270] T b6c53e621b464de98f002df9b8059689 P 285843747942468fbf40a845620eae2a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.924219 12636 tablet_replica.cc:331] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.924667 12636 raft_consensus.cc:2241] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 4 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.925303 12636 raft_consensus.cc:2270] T d51ca69a4cca450fb24989f8eeef180d P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.926791 12636 tablet_replica.cc:331] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a: stopping tablet replica
I20250623 01:58:10.927214 12636 raft_consensus.cc:2241] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:10.927588 12636 raft_consensus.cc:2270] T 13009913457c422c90edf7642617a9a6 P 285843747942468fbf40a845620eae2a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.943776 12636 tablet_server.cc:195] TabletServer@127.12.87.3:0 shutdown complete.
I20250623 01:58:10.961628 12636 master.cc:561] Master@127.12.87.62:45311 shutting down...
I20250623 01:58:10.981249 12636 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 1 LEADER]: Raft consensus shutting down.
I20250623 01:58:10.981753 12636 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:10.982019 12636 tablet_replica.cc:331] T 00000000000000000000000000000000 P a4a0ff7480744f68b50ddd187eb4351d: stopping tablet replica
I20250623 01:58:11.000895 12636 master.cc:583] Master@127.12.87.62:45311 shutdown complete.
05ca19107f7b442bb011af4486c39121 4
285843747942468fbf40a845620eae2a 1
61244692ea674370a1678cee534ef92f 5
05ca19107f7b442bb011af4486c39121 4
285843747942468fbf40a845620eae2a 3
61244692ea674370a1678cee534ef92f 3
05ca19107f7b442bb011af4486c39121 1
285843747942468fbf40a845620eae2a 1
61244692ea674370a1678cee534ef92f 8
05ca19107f7b442bb011af4486c39121 3
285843747942468fbf40a845620eae2a 3
61244692ea674370a1678cee534ef92f 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28561 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250623 01:58:11.039783 12636 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.87.62:45405
I20250623 01:58:11.040848 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:11.046355 13685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:11.046803 13688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:11.046994 13686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:11.048331 12636 server_base.cc:1048] running on GCE node
I20250623 01:58:11.049172 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:58:11.049391 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:58:11.049575 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643891049558 us; error 0 us; skew 500 ppm
I20250623 01:58:11.050117 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:11.052399 12636 webserver.cc:469] Webserver started at http://127.12.87.62:40719/ using document root <none> and password file <none>
I20250623 01:58:11.052825 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:11.052971 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:11.053190 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:11.054292 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/master-0-root/instance:
uuid: "bc0f35fa176545c3a95a38f727af3536"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-47g2"
I20250623 01:58:11.058403 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250623 01:58:11.061494 13693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.062165 12636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250623 01:58:11.062422 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/master-0-root
uuid: "bc0f35fa176545c3a95a38f727af3536"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-47g2"
I20250623 01:58:11.062680 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:11.086304 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:11.087967 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:11.127943 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.62:45405
I20250623 01:58:11.128103 13745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.62:45405 every 8 connection(s)
I20250623 01:58:11.132174 13746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.142015 13746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Bootstrap starting.
I20250623 01:58:11.146636 13746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:11.150784 13746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: No bootstrap required, opened a new log
I20250623 01:58:11.152904 13746 raft_consensus.cc:357] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f35fa176545c3a95a38f727af3536" member_type: VOTER }
I20250623 01:58:11.153409 13746 raft_consensus.cc:383] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:11.153620 13746 raft_consensus.cc:738] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0f35fa176545c3a95a38f727af3536, State: Initialized, Role: FOLLOWER
I20250623 01:58:11.154131 13746 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f35fa176545c3a95a38f727af3536" member_type: VOTER }
I20250623 01:58:11.154549 13746 raft_consensus.cc:397] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:11.154726 13746 raft_consensus.cc:491] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:11.154939 13746 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:11.158885 13746 raft_consensus.cc:513] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f35fa176545c3a95a38f727af3536" member_type: VOTER }
I20250623 01:58:11.159307 13746 leader_election.cc:304] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc0f35fa176545c3a95a38f727af3536; no voters:
I20250623 01:58:11.160318 13746 leader_election.cc:290] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:58:11.160714 13749 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:11.161960 13749 raft_consensus.cc:695] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [term 1 LEADER]: Becoming Leader. State: Replica: bc0f35fa176545c3a95a38f727af3536, State: Running, Role: LEADER
I20250623 01:58:11.162564 13749 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f35fa176545c3a95a38f727af3536" member_type: VOTER }
I20250623 01:58:11.163256 13746 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:58:11.169516 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc0f35fa176545c3a95a38f727af3536. Latest consensus state: current_term: 1 leader_uuid: "bc0f35fa176545c3a95a38f727af3536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f35fa176545c3a95a38f727af3536" member_type: VOTER } }
I20250623 01:58:11.169416 13750 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc0f35fa176545c3a95a38f727af3536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f35fa176545c3a95a38f727af3536" member_type: VOTER } }
I20250623 01:58:11.170172 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:11.170230 13750 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:11.172224 13758 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:58:11.177736 13758 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:58:11.178772 12636 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250623 01:58:11.185660 13758 catalog_manager.cc:1349] Generated new cluster ID: 841c4f756af24e42a463e619a7295fc3
I20250623 01:58:11.185956 13758 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:58:11.206775 13758 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:58:11.207943 13758 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:58:11.221764 13758 catalog_manager.cc:5955] T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Generated new TSK 0
I20250623 01:58:11.222338 13758 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:58:11.245225 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:11.251609 13767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:11.252820 13768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:11.254468 13770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:11.254878 12636 server_base.cc:1048] running on GCE node
I20250623 01:58:11.255705 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:58:11.255923 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:58:11.256063 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643891256048 us; error 0 us; skew 500 ppm
I20250623 01:58:11.256544 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:11.258874 12636 webserver.cc:469] Webserver started at http://127.12.87.1:39357/ using document root <none> and password file <none>
I20250623 01:58:11.259320 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:11.259505 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:11.259779 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:11.260969 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root/instance:
uuid: "85a58a834b6b4b0dbe197b3dad87e382"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-47g2"
I20250623 01:58:11.265848 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250623 01:58:11.269105 13775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.269824 12636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250623 01:58:11.270099 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root
uuid: "85a58a834b6b4b0dbe197b3dad87e382"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-47g2"
I20250623 01:58:11.270360 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:11.287339 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:11.288480 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:11.289942 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:11.292253 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:11.292466 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.292688 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:11.292850 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.330740 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.1:34781
I20250623 01:58:11.330840 13837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.1:34781 every 8 connection(s)
I20250623 01:58:11.335287 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:11.343248 13842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:11.345624 13843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:11.346302 12636 server_base.cc:1048] running on GCE node
W20250623 01:58:11.348098 13845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:11.348968 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250623 01:58:11.349079 13838 heartbeater.cc:344] Connected to a master server at 127.12.87.62:45405
W20250623 01:58:11.349174 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:58:11.349491 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643891349470 us; error 0 us; skew 500 ppm
I20250623 01:58:11.349550 13838 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:11.350065 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:11.350286 13838 heartbeater.cc:507] Master 127.12.87.62:45405 requested a full tablet report, sending...
I20250623 01:58:11.352936 12636 webserver.cc:469] Webserver started at http://127.12.87.2:36027/ using document root <none> and password file <none>
I20250623 01:58:11.352968 13711 ts_manager.cc:194] Registered new tserver with Master: 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:11.353678 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:11.353940 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:11.354245 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:11.355433 13711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52432
I20250623 01:58:11.355742 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/instance:
uuid: "ee041a8325864e788a383b8356afc652"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-47g2"
I20250623 01:58:11.360086 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250623 01:58:11.363099 13850 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.363732 12636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250623 01:58:11.364001 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root
uuid: "ee041a8325864e788a383b8356afc652"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-47g2"
I20250623 01:58:11.364241 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:11.378361 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:11.379477 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:11.380812 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:11.382912 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:11.383100 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.383318 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:11.383456 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.423161 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.2:43055
I20250623 01:58:11.423249 13912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.2:43055 every 8 connection(s)
I20250623 01:58:11.427476 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:11.433846 13916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:11.436844 13917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:11.437451 12636 server_base.cc:1048] running on GCE node
W20250623 01:58:11.438197 13919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:11.439093 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250623 01:58:11.439150 13913 heartbeater.cc:344] Connected to a master server at 127.12.87.62:45405
W20250623 01:58:11.439309 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:58:11.439503 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643891439488 us; error 0 us; skew 500 ppm
I20250623 01:58:11.439523 13913 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:11.440073 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:11.440265 13913 heartbeater.cc:507] Master 127.12.87.62:45405 requested a full tablet report, sending...
I20250623 01:58:11.442276 13711 ts_manager.cc:194] Registered new tserver with Master: ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:11.442878 12636 webserver.cc:469] Webserver started at http://127.12.87.3:40355/ using document root <none> and password file <none>
I20250623 01:58:11.443333 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:11.443521 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:11.443733 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:11.443675 13711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52438
I20250623 01:58:11.445003 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/instance:
uuid: "c243715fe12445019c33bbeafbaea635"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-47g2"
I20250623 01:58:11.449084 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250623 01:58:11.452307 13924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.452958 12636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250623 01:58:11.453214 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root
uuid: "c243715fe12445019c33bbeafbaea635"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-47g2"
I20250623 01:58:11.453500 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:11.470999 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:11.472231 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:11.473724 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:11.475889 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:11.476082 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.476289 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:11.476436 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.513950 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.3:33369
I20250623 01:58:11.514045 13986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.3:33369 every 8 connection(s)
I20250623 01:58:11.526451 13987 heartbeater.cc:344] Connected to a master server at 127.12.87.62:45405
I20250623 01:58:11.526813 13987 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:11.527503 13987 heartbeater.cc:507] Master 127.12.87.62:45405 requested a full tablet report, sending...
I20250623 01:58:11.529146 13711 ts_manager.cc:194] Registered new tserver with Master: c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:11.530148 12636 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013037737s
I20250623 01:58:11.530403 13711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52444
I20250623 01:58:11.532579 12636 test_util.cc:276] Using random seed: -1958186224
I20250623 01:58:11.554359 13711 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52460:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 01:58:11.560166 13711 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 01:58:11.755373 13952 tablet_service.cc:1468] Processing CreateTablet for tablet 2d3844327a1b452da89d70d856d0a80f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250623 01:58:11.755640 13951 tablet_service.cc:1468] Processing CreateTablet for tablet d2a94c049f124a68ad6678cff7f838ac (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250623 01:58:11.756536 13803 tablet_service.cc:1468] Processing CreateTablet for tablet 2d3844327a1b452da89d70d856d0a80f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250623 01:58:11.756863 13801 tablet_service.cc:1468] Processing CreateTablet for tablet afad1ff768904b11936d338d33df7998 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250623 01:58:11.758059 13803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3844327a1b452da89d70d856d0a80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.757782 13950 tablet_service.cc:1468] Processing CreateTablet for tablet afad1ff768904b11936d338d33df7998 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250623 01:58:11.758682 13801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afad1ff768904b11936d338d33df7998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.758572 13948 tablet_service.cc:1468] Processing CreateTablet for tablet 5d4cc11489de46dcb7d5f77daefd49af (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250623 01:58:11.757139 13952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3844327a1b452da89d70d856d0a80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.760046 13949 tablet_service.cc:1468] Processing CreateTablet for tablet 939deead84d54fe0b858c13a3dfaf686 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250623 01:58:11.761407 13949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939deead84d54fe0b858c13a3dfaf686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.762554 13802 tablet_service.cc:1468] Processing CreateTablet for tablet d2a94c049f124a68ad6678cff7f838ac (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250623 01:58:11.763911 13802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a94c049f124a68ad6678cff7f838ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.764122 13951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a94c049f124a68ad6678cff7f838ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.766063 13800 tablet_service.cc:1468] Processing CreateTablet for tablet 939deead84d54fe0b858c13a3dfaf686 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250623 01:58:11.768272 13799 tablet_service.cc:1468] Processing CreateTablet for tablet 5d4cc11489de46dcb7d5f77daefd49af (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250623 01:58:11.769635 13948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4cc11489de46dcb7d5f77daefd49af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.771972 13950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afad1ff768904b11936d338d33df7998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.772158 13800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939deead84d54fe0b858c13a3dfaf686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.773702 13799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4cc11489de46dcb7d5f77daefd49af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.779458 13798 tablet_service.cc:1468] Processing CreateTablet for tablet de00fb2e24074586bfe02d33fb92e4e6 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250623 01:58:11.780910 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de00fb2e24074586bfe02d33fb92e4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.792553 13947 tablet_service.cc:1468] Processing CreateTablet for tablet de00fb2e24074586bfe02d33fb92e4e6 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250623 01:58:11.794047 13947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de00fb2e24074586bfe02d33fb92e4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.798564 13797 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb699d5420143efa03140964d5aa739 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250623 01:58:11.800037 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb699d5420143efa03140964d5aa739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.802431 13796 tablet_service.cc:1468] Processing CreateTablet for tablet 272c03e6252d489cbf2fdceb36b05dc1 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250623 01:58:11.803788 13796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272c03e6252d489cbf2fdceb36b05dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.804629 13872 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb699d5420143efa03140964d5aa739 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250623 01:58:11.806461 13871 tablet_service.cc:1468] Processing CreateTablet for tablet 272c03e6252d489cbf2fdceb36b05dc1 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250623 01:58:11.807898 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272c03e6252d489cbf2fdceb36b05dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.807648 13876 tablet_service.cc:1468] Processing CreateTablet for tablet afad1ff768904b11936d338d33df7998 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250623 01:58:11.809015 13876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afad1ff768904b11936d338d33df7998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.809998 13870 tablet_service.cc:1468] Processing CreateTablet for tablet 50c92eb44c6c480d832145fd7c9c8d3f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250623 01:58:11.810473 13875 tablet_service.cc:1468] Processing CreateTablet for tablet 939deead84d54fe0b858c13a3dfaf686 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250623 01:58:11.812253 13869 tablet_service.cc:1468] Processing CreateTablet for tablet 32faa630311f4d8e99b289422fe771f5 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250623 01:58:11.812624 13874 tablet_service.cc:1468] Processing CreateTablet for tablet 5d4cc11489de46dcb7d5f77daefd49af (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250623 01:58:11.806457 13877 tablet_service.cc:1468] Processing CreateTablet for tablet d2a94c049f124a68ad6678cff7f838ac (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250623 01:58:11.814806 13873 tablet_service.cc:1468] Processing CreateTablet for tablet de00fb2e24074586bfe02d33fb92e4e6 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250623 01:58:11.817117 13878 tablet_service.cc:1468] Processing CreateTablet for tablet 2d3844327a1b452da89d70d856d0a80f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250623 01:58:11.819362 13795 tablet_service.cc:1468] Processing CreateTablet for tablet 50c92eb44c6c480d832145fd7c9c8d3f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250623 01:58:11.820741 13795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c92eb44c6c480d832145fd7c9c8d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.822472 13877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a94c049f124a68ad6678cff7f838ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.822695 13794 tablet_service.cc:1468] Processing CreateTablet for tablet 32faa630311f4d8e99b289422fe771f5 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250623 01:58:11.823113 13875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939deead84d54fe0b858c13a3dfaf686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.823810 13874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4cc11489de46dcb7d5f77daefd49af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.824026 13794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32faa630311f4d8e99b289422fe771f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.827375 13870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c92eb44c6c480d832145fd7c9c8d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.829625 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32faa630311f4d8e99b289422fe771f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.832151 13946 tablet_service.cc:1468] Processing CreateTablet for tablet 272c03e6252d489cbf2fdceb36b05dc1 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250623 01:58:11.833724 13946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272c03e6252d489cbf2fdceb36b05dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.833499 13949 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb699d5420143efa03140964d5aa739 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250623 01:58:11.834714 13949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb699d5420143efa03140964d5aa739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.834463 13943 tablet_service.cc:1468] Processing CreateTablet for tablet b9e5783d0e1d49b899145b9d9e01f6f9 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250623 01:58:11.833113 13872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb699d5420143efa03140964d5aa739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.835785 13943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e5783d0e1d49b899145b9d9e01f6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.835940 13873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de00fb2e24074586bfe02d33fb92e4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.838512 13944 tablet_service.cc:1468] Processing CreateTablet for tablet 32faa630311f4d8e99b289422fe771f5 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250623 01:58:11.839840 13944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32faa630311f4d8e99b289422fe771f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.841639 13878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3844327a1b452da89d70d856d0a80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.844324 13945 tablet_service.cc:1468] Processing CreateTablet for tablet 50c92eb44c6c480d832145fd7c9c8d3f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250623 01:58:11.845626 13945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c92eb44c6c480d832145fd7c9c8d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.861083 14003 tablet_bootstrap.cc:492] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:11.882627 13952 tablet_service.cc:1468] Processing CreateTablet for tablet 3828994fa152441a95c5034d002516db (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250623 01:58:11.897913 13952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3828994fa152441a95c5034d002516db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.900151 13798 tablet_service.cc:1468] Processing CreateTablet for tablet 3828994fa152441a95c5034d002516db (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250623 01:58:11.908176 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3828994fa152441a95c5034d002516db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.902263 13951 tablet_service.cc:1468] Processing CreateTablet for tablet 433978dce7d94bcb8968bc09583357c4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250623 01:58:11.922645 13951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 433978dce7d94bcb8968bc09583357c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.916110 13800 tablet_service.cc:1468] Processing CreateTablet for tablet 433978dce7d94bcb8968bc09583357c4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250623 01:58:11.927328 13796 tablet_service.cc:1468] Processing CreateTablet for tablet 819094ffb4314769ab2a06512232fc61 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250623 01:58:11.927909 13800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 433978dce7d94bcb8968bc09583357c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.928465 13796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819094ffb4314769ab2a06512232fc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.913071 14003 tablet_bootstrap.cc:654] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:11.915043 13802 tablet_service.cc:1468] Processing CreateTablet for tablet bdaf5d40df984f96b90a4d8ace220223 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250623 01:58:11.900909 13874 tablet_service.cc:1468] Processing CreateTablet for tablet b9e5783d0e1d49b899145b9d9e01f6f9 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250623 01:58:11.915953 13799 tablet_service.cc:1468] Processing CreateTablet for tablet 02e07b6deb3546fcad55d41d2ecd8a5a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250623 01:58:11.941030 13874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e5783d0e1d49b899145b9d9e01f6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.940796 13870 tablet_service.cc:1468] Processing CreateTablet for tablet 3828994fa152441a95c5034d002516db (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250623 01:58:11.941031 13802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaf5d40df984f96b90a4d8ace220223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.942060 13870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3828994fa152441a95c5034d002516db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.942746 13947 tablet_service.cc:1468] Processing CreateTablet for tablet 02e07b6deb3546fcad55d41d2ecd8a5a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250623 01:58:11.943135 13950 tablet_service.cc:1468] Processing CreateTablet for tablet 819094ffb4314769ab2a06512232fc61 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250623 01:58:11.943845 13947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e07b6deb3546fcad55d41d2ecd8a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.947398 13799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e07b6deb3546fcad55d41d2ecd8a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.951442 13871 tablet_service.cc:1468] Processing CreateTablet for tablet bdaf5d40df984f96b90a4d8ace220223 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250623 01:58:11.952659 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaf5d40df984f96b90a4d8ace220223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.953951 13801 tablet_service.cc:1468] Processing CreateTablet for tablet 5d1d49d7480d4088be0d5ed6baae9489 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250623 01:58:11.955190 13801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d1d49d7480d4088be0d5ed6baae9489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.946436 13950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819094ffb4314769ab2a06512232fc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.959324 13945 tablet_service.cc:1468] Processing CreateTablet for tablet 5d1d49d7480d4088be0d5ed6baae9489 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250623 01:58:11.960481 13945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d1d49d7480d4088be0d5ed6baae9489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.967773 13874 tablet_service.cc:1468] Processing CreateTablet for tablet 02e07b6deb3546fcad55d41d2ecd8a5a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250623 01:58:11.968899 13874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e07b6deb3546fcad55d41d2ecd8a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.970692 13795 tablet_service.cc:1468] Processing CreateTablet for tablet 839cb616e68044308dfa7e08eadc1903 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250623 01:58:11.978426 13872 tablet_service.cc:1468] Processing CreateTablet for tablet 819094ffb4314769ab2a06512232fc61 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250623 01:58:11.889201 13803 tablet_service.cc:1468] Processing CreateTablet for tablet b9e5783d0e1d49b899145b9d9e01f6f9 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250623 01:58:11.983623 13943 tablet_service.cc:1468] Processing CreateTablet for tablet 839cb616e68044308dfa7e08eadc1903 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250623 01:58:12.014009 13943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839cb616e68044308dfa7e08eadc1903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.014137 13874 tablet_service.cc:1468] Processing CreateTablet for tablet 933266ad75974c449e00b527ab278f8f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250623 01:58:11.920367 13948 tablet_service.cc:1468] Processing CreateTablet for tablet bdaf5d40df984f96b90a4d8ace220223 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250623 01:58:11.989745 14002 tablet_bootstrap.cc:492] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:11.994359 13794 tablet_service.cc:1468] Processing CreateTablet for tablet 92909e3375c549b5baa0595e1750fa1f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250623 01:58:11.999342 13876 tablet_service.cc:1468] Processing CreateTablet for tablet 433978dce7d94bcb8968bc09583357c4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250623 01:58:12.000499 13877 tablet_service.cc:1468] Processing CreateTablet for tablet 5d1d49d7480d4088be0d5ed6baae9489 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250623 01:58:12.001883 13797 tablet_service.cc:1468] Processing CreateTablet for tablet 933266ad75974c449e00b527ab278f8f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250623 01:58:12.011368 13872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819094ffb4314769ab2a06512232fc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.011899 13800 tablet_service.cc:1468] Processing CreateTablet for tablet 2540092c63cb4d0cbe07ae4018f99f46 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250623 01:58:12.012666 13944 tablet_service.cc:1468] Processing CreateTablet for tablet 933266ad75974c449e00b527ab278f8f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250623 01:58:12.017946 13948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaf5d40df984f96b90a4d8ace220223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.028268 14004 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:12.030709 13876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 433978dce7d94bcb8968bc09583357c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.032754 13878 tablet_service.cc:1468] Processing CreateTablet for tablet 92909e3375c549b5baa0595e1750fa1f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250623 01:58:12.033636 13877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d1d49d7480d4088be0d5ed6baae9489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.034322 13944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933266ad75974c449e00b527ab278f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.034924 13875 tablet_service.cc:1468] Processing CreateTablet for tablet 2540092c63cb4d0cbe07ae4018f99f46 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250623 01:58:12.035375 13949 tablet_service.cc:1468] Processing CreateTablet for tablet 92909e3375c549b5baa0595e1750fa1f (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250623 01:58:12.019652 13873 tablet_service.cc:1468] Processing CreateTablet for tablet 839cb616e68044308dfa7e08eadc1903 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250623 01:58:12.036906 13946 tablet_service.cc:1468] Processing CreateTablet for tablet 2540092c63cb4d0cbe07ae4018f99f46 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250623 01:58:12.039458 14002 tablet_bootstrap.cc:654] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.050978 13874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933266ad75974c449e00b527ab278f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.056015 13949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92909e3375c549b5baa0595e1750fa1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.063840 13802 tablet_service.cc:1468] Processing CreateTablet for tablet e773c5700b0945d1981f481534952e85 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250623 01:58:12.067435 13802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e773c5700b0945d1981f481534952e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.067243 13799 tablet_service.cc:1468] Processing CreateTablet for tablet ade11e463c53492f8c17cba4e0500ec4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250623 01:58:12.071835 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933266ad75974c449e00b527ab278f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.072438 13800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2540092c63cb4d0cbe07ae4018f99f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.073768 13794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92909e3375c549b5baa0595e1750fa1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.074419 13948 tablet_service.cc:1468] Processing CreateTablet for tablet cd5ada8676314a73af7cef4c5cd5553e (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250623 01:58:12.075556 13948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5ada8676314a73af7cef4c5cd5553e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.077728 13803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e5783d0e1d49b899145b9d9e01f6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.078368 13795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839cb616e68044308dfa7e08eadc1903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.086122 13878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92909e3375c549b5baa0595e1750fa1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.091176 13946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2540092c63cb4d0cbe07ae4018f99f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.105791 13873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839cb616e68044308dfa7e08eadc1903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.106855 14003 tablet_bootstrap.cc:492] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:12.107327 14003 ts_tablet_manager.cc:1397] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.247s user 0.013s sys 0.032s
I20250623 01:58:12.107707 13875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2540092c63cb4d0cbe07ae4018f99f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.115341 13870 tablet_service.cc:1468] Processing CreateTablet for tablet ade11e463c53492f8c17cba4e0500ec4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250623 01:58:12.116521 13870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade11e463c53492f8c17cba4e0500ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.117455 13869 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa785c3afe64dbcac3b658b49195b8c (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250623 01:58:12.118592 13950 tablet_service.cc:1468] Processing CreateTablet for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250623 01:58:12.119740 13950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.120218 13947 tablet_service.cc:1468] Processing CreateTablet for tablet ade11e463c53492f8c17cba4e0500ec4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250623 01:58:12.121528 13947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade11e463c53492f8c17cba4e0500ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.122586 13952 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa785c3afe64dbcac3b658b49195b8c (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250623 01:58:12.123168 14003 raft_consensus.cc:357] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.123735 13952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa785c3afe64dbcac3b658b49195b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.123888 14003 raft_consensus.cc:383] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.124223 14003 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.125829 13799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade11e463c53492f8c17cba4e0500ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.126542 13876 tablet_service.cc:1468] Processing CreateTablet for tablet cd5ada8676314a73af7cef4c5cd5553e (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250623 01:58:12.128399 13801 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa785c3afe64dbcac3b658b49195b8c (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250623 01:58:12.129586 13801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa785c3afe64dbcac3b658b49195b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.130609 13951 tablet_service.cc:1468] Processing CreateTablet for tablet 1b3574beac134b2eabbd65a4f54b3ab3 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250623 01:58:12.131829 13951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b3574beac134b2eabbd65a4f54b3ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.134672 14003 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.137418 13943 tablet_service.cc:1468] Processing CreateTablet for tablet 7895c36aec9944b7981b0a0ccacc6556 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250623 01:58:12.135769 13877 tablet_service.cc:1468] Processing CreateTablet for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250623 01:58:12.138731 13943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7895c36aec9944b7981b0a0ccacc6556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.132779 13945 tablet_service.cc:1468] Processing CreateTablet for tablet e773c5700b0945d1981f481534952e85 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250623 01:58:12.113857 13871 tablet_service.cc:1468] Processing CreateTablet for tablet e773c5700b0945d1981f481534952e85 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250623 01:58:12.118606 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa785c3afe64dbcac3b658b49195b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.144085 13874 tablet_service.cc:1468] Processing CreateTablet for tablet 1b3574beac134b2eabbd65a4f54b3ab3 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250623 01:58:12.145179 13874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b3574beac134b2eabbd65a4f54b3ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.150207 13798 tablet_service.cc:1468] Processing CreateTablet for tablet 1b3574beac134b2eabbd65a4f54b3ab3 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250623 01:58:12.151401 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b3574beac134b2eabbd65a4f54b3ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.155058 13945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e773c5700b0945d1981f481534952e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.155463 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e773c5700b0945d1981f481534952e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.161595 13876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5ada8676314a73af7cef4c5cd5553e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 01:58:12.174062 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:12.176122 13795 tablet_service.cc:1468] Processing CreateTablet for tablet c708cd61006343f09750b88b2621eec6 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250623 01:58:12.164877 14004 tablet_bootstrap.cc:654] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.174268 13878 tablet_service.cc:1468] Processing CreateTablet for tablet 7895c36aec9944b7981b0a0ccacc6556 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250623 01:58:12.164893 13794 tablet_service.cc:1468] Processing CreateTablet for tablet 7895c36aec9944b7981b0a0ccacc6556 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250623 01:58:12.136111 13796 tablet_service.cc:1468] Processing CreateTablet for tablet cd5ada8676314a73af7cef4c5cd5553e (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250623 01:58:12.180931 13797 tablet_service.cc:1468] Processing CreateTablet for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250623 01:58:12.165571 13948 tablet_service.cc:1468] Processing CreateTablet for tablet c708cd61006343f09750b88b2621eec6 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250623 01:58:12.188051 13877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.192555 13878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7895c36aec9944b7981b0a0ccacc6556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.193964 13948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c708cd61006343f09750b88b2621eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.194859 14002 tablet_bootstrap.cc:492] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:12.195364 14002 ts_tablet_manager.cc:1397] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.206s user 0.014s sys 0.005s
I20250623 01:58:12.198119 14002 raft_consensus.cc:357] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.187583 13795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c708cd61006343f09750b88b2621eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.199443 13949 tablet_service.cc:1468] Processing CreateTablet for tablet f023b481774a4d06bf5c5c62a3df3012 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250623 01:58:12.190299 13944 tablet_service.cc:1468] Processing CreateTablet for tablet adb1e25eef1743848cf218e294bc9226 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250623 01:58:12.215672 13949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f023b481774a4d06bf5c5c62a3df3012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.207391 14002 raft_consensus.cc:383] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.218905 13800 tablet_service.cc:1468] Processing CreateTablet for tablet adb1e25eef1743848cf218e294bc9226 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250623 01:58:12.223227 14002 raft_consensus.cc:738] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.225993 13876 tablet_service.cc:1468] Processing CreateTablet for tablet adb1e25eef1743848cf218e294bc9226 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250623 01:58:12.225898 14002 consensus_queue.cc:260] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.227207 13876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb1e25eef1743848cf218e294bc9226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.232726 13875 tablet_service.cc:1468] Processing CreateTablet for tablet f023b481774a4d06bf5c5c62a3df3012 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250623 01:58:12.237735 13987 heartbeater.cc:499] Master 127.12.87.62:45405 was elected leader, sending a full tablet report...
I20250623 01:58:12.238835 13873 tablet_service.cc:1468] Processing CreateTablet for tablet 318e5309795f47f3868e0787ddfdd146 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250623 01:58:12.240057 13873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 318e5309795f47f3868e0787ddfdd146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.241933 13875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f023b481774a4d06bf5c5c62a3df3012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.237603 14007 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:12.244287 14007 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.222553 13794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7895c36aec9944b7981b0a0ccacc6556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.245868 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.246964 13872 tablet_service.cc:1468] Processing CreateTablet for tablet c708cd61006343f09750b88b2621eec6 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250623 01:58:12.248108 13872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c708cd61006343f09750b88b2621eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.244746 13946 tablet_service.cc:1468] Processing CreateTablet for tablet 318e5309795f47f3868e0787ddfdd146 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250623 01:58:12.249867 14003 ts_tablet_manager.cc:1428] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.142s user 0.006s sys 0.005s
I20250623 01:58:12.250378 13796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5ada8676314a73af7cef4c5cd5553e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.250408 13946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 318e5309795f47f3868e0787ddfdd146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.256870 14007 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:12.265110 14004 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:12.265606 14004 ts_tablet_manager.cc:1397] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.238s user 0.005s sys 0.008s
I20250623 01:58:12.269219 14003 tablet_bootstrap.cc:492] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:12.277531 14004 raft_consensus.cc:357] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.270983 13800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb1e25eef1743848cf218e294bc9226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.294890 13838 heartbeater.cc:499] Master 127.12.87.62:45405 was elected leader, sending a full tablet report...
I20250623 01:58:12.295359 14002 ts_tablet_manager.cc:1428] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.100s user 0.019s sys 0.025s
I20250623 01:58:12.308679 13944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb1e25eef1743848cf218e294bc9226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.312733 13871 tablet_service.cc:1468] Processing CreateTablet for tablet 48dd5f33321544e58bc600dbff358693 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250623 01:58:12.313695 13799 tablet_service.cc:1468] Processing CreateTablet for tablet 318e5309795f47f3868e0787ddfdd146 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250623 01:58:12.314186 13877 tablet_service.cc:1468] Processing CreateTablet for tablet a71ef18fd45046578f969e8e8905e897 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250623 01:58:12.315937 13803 tablet_service.cc:1468] Processing CreateTablet for tablet a71ef18fd45046578f969e8e8905e897 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250623 01:58:12.305588 14004 raft_consensus.cc:383] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.334849 13877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a71ef18fd45046578f969e8e8905e897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.334769 14004 raft_consensus.cc:738] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.335542 14004 consensus_queue.cc:260] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.317248 14003 tablet_bootstrap.cc:654] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.320844 13947 tablet_service.cc:1468] Processing CreateTablet for tablet 48dd5f33321544e58bc600dbff358693 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250623 01:58:12.338452 13947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48dd5f33321544e58bc600dbff358693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.323465 13798 tablet_service.cc:1468] Processing CreateTablet for tablet 48dd5f33321544e58bc600dbff358693 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250623 01:58:12.341152 13878 tablet_service.cc:1468] Processing CreateTablet for tablet e2928af9e5df4f9eacf59f499ab96dc1 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250623 01:58:12.349530 13878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2928af9e5df4f9eacf59f499ab96dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.355464 13801 tablet_service.cc:1468] Processing CreateTablet for tablet e2928af9e5df4f9eacf59f499ab96dc1 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250623 01:58:12.344987 13945 tablet_service.cc:1468] Processing CreateTablet for tablet ff8dea72f9f9447abaa5fa803f5e84e8 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250623 01:58:12.358462 13945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8dea72f9f9447abaa5fa803f5e84e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.324656 14002 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:12.366851 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48dd5f33321544e58bc600dbff358693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.326795 13797 tablet_service.cc:1468] Processing CreateTablet for tablet ff8dea72f9f9447abaa5fa803f5e84e8 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250623 01:58:12.330533 13869 tablet_service.cc:1468] Processing CreateTablet for tablet ff8dea72f9f9447abaa5fa803f5e84e8 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250623 01:58:12.267539 13802 tablet_service.cc:1468] Processing CreateTablet for tablet f023b481774a4d06bf5c5c62a3df3012 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250623 01:58:12.370222 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8dea72f9f9447abaa5fa803f5e84e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.331825 13950 tablet_service.cc:1468] Processing CreateTablet for tablet a71ef18fd45046578f969e8e8905e897 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250623 01:58:12.371893 13950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a71ef18fd45046578f969e8e8905e897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.333979 13799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 318e5309795f47f3868e0787ddfdd146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.373097 13801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2928af9e5df4f9eacf59f499ab96dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.374193 13802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f023b481774a4d06bf5c5c62a3df3012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.382717 13803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a71ef18fd45046578f969e8e8905e897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.386998 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8dea72f9f9447abaa5fa803f5e84e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.387797 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48dd5f33321544e58bc600dbff358693. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 01:58:12.406301 13697 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 60665.2us
I20250623 01:58:12.407155 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:12.407619 13694 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 60350.9us
W20250623 01:58:12.408856 13925 leader_election.cc:343] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:12.445886 13913 heartbeater.cc:499] Master 127.12.87.62:45405 was elected leader, sending a full tablet report...
I20250623 01:58:12.455365 13800 tablet_service.cc:1468] Processing CreateTablet for tablet b39b05807ed544d388835cc364cb7274 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250623 01:58:12.456701 13800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b39b05807ed544d388835cc364cb7274. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.461247 13944 tablet_service.cc:1468] Processing CreateTablet for tablet e2928af9e5df4f9eacf59f499ab96dc1 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250623 01:58:12.462544 13944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2928af9e5df4f9eacf59f499ab96dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.482741 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:12.484323 13926 leader_election.cc:343] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:12.480829 13874 tablet_service.cc:1468] Processing CreateTablet for tablet b39b05807ed544d388835cc364cb7274 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250623 01:58:12.484774 13926 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:12.496140 14009 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:12.502624 14010 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:12.502964 14009 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.484818 13876 tablet_service.cc:1468] Processing CreateTablet for tablet 0f12346da5a14097bbfa150cb8d1d907 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250623 01:58:12.509189 14009 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:12.530881 14004 ts_tablet_manager.cc:1428] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.265s user 0.006s sys 0.019s
I20250623 01:58:12.538533 14002 tablet_bootstrap.cc:654] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.540294 13951 tablet_service.cc:1468] Processing CreateTablet for tablet 6f13ab3fda6f4c6b9b68d33e7a89c988 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250623 01:58:12.534667 13795 tablet_service.cc:1468] Processing CreateTablet for tablet 0f12346da5a14097bbfa150cb8d1d907 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250623 01:58:12.543471 13951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f13ab3fda6f4c6b9b68d33e7a89c988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.543900 13795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f12346da5a14097bbfa150cb8d1d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.545354 13870 tablet_service.cc:1468] Processing CreateTablet for tablet ba9a759dbf744a06bcb77355efdf4085 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250623 01:58:12.546932 13872 tablet_service.cc:1468] Processing CreateTablet for tablet 6f13ab3fda6f4c6b9b68d33e7a89c988 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250623 01:58:12.547937 14004 tablet_bootstrap.cc:492] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:12.550812 13946 tablet_service.cc:1468] Processing CreateTablet for tablet ba9a759dbf744a06bcb77355efdf4085 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250623 01:58:12.560379 13877 tablet_service.cc:1468] Processing CreateTablet for tablet 0c8353de982c4260a5ecf1445df4ba21 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250623 01:58:12.560941 13875 tablet_service.cc:1468] Processing CreateTablet for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250623 01:58:12.572816 13946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9a759dbf744a06bcb77355efdf4085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.578426 13952 tablet_service.cc:1468] Processing CreateTablet for tablet 0c8353de982c4260a5ecf1445df4ba21 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250623 01:58:12.579684 13952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8353de982c4260a5ecf1445df4ba21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.541440 13874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b39b05807ed544d388835cc364cb7274. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.547938 13948 tablet_service.cc:1468] Processing CreateTablet for tablet 0f12346da5a14097bbfa150cb8d1d907 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250623 01:58:12.583632 13948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f12346da5a14097bbfa150cb8d1d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.584362 13876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f12346da5a14097bbfa150cb8d1d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.585623 13875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.586248 13872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f13ab3fda6f4c6b9b68d33e7a89c988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.530612 13943 tablet_service.cc:1468] Processing CreateTablet for tablet b39b05807ed544d388835cc364cb7274 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250623 01:58:12.588788 13943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b39b05807ed544d388835cc364cb7274. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.587817 13870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9a759dbf744a06bcb77355efdf4085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.592451 14003 tablet_bootstrap.cc:492] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:12.592947 14003 ts_tablet_manager.cc:1397] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.336s user 0.018s sys 0.009s
I20250623 01:58:12.595742 14003 raft_consensus.cc:357] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:12.596419 14003 raft_consensus.cc:383] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.596704 14003 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.598958 13873 tablet_service.cc:1468] Processing CreateTablet for tablet c078b30fc50b475193ae485fbe10e04c (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250623 01:58:12.606140 13877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8353de982c4260a5ecf1445df4ba21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.611176 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:12.612653 13776 leader_election.cc:343] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:12.614118 13949 tablet_service.cc:1468] Processing CreateTablet for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250623 01:58:12.615085 13949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.601614 14003 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:12.623271 13799 tablet_service.cc:1468] Processing CreateTablet for tablet ba9a759dbf744a06bcb77355efdf4085 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250623 01:58:12.624472 13799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9a759dbf744a06bcb77355efdf4085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.628371 14003 ts_tablet_manager.cc:1428] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.035s user 0.007s sys 0.001s
I20250623 01:58:12.629390 14003 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:12.639336 13952 tablet_service.cc:1468] Processing CreateTablet for tablet c078b30fc50b475193ae485fbe10e04c (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250623 01:58:12.641568 13952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c078b30fc50b475193ae485fbe10e04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.659431 14010 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:12.659976 14010 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:12.662056 14010 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:12.664063 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:12.665998 13878 tablet_service.cc:1468] Processing CreateTablet for tablet 7560a94e265b433dbc427ef13c120669 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250623 01:58:12.666713 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:12.669011 13925 leader_election.cc:343] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
W20250623 01:58:12.670126 13926 leader_election.cc:343] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:12.670537 13796 tablet_service.cc:1468] Processing CreateTablet for tablet 0c8353de982c4260a5ecf1445df4ba21 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250623 01:58:12.670539 13926 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:12.671768 13796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8353de982c4260a5ecf1445df4ba21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.671628 13951 tablet_service.cc:1468] Processing CreateTablet for tablet 632bb79cb33a491fa5765cd80357f1a4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250623 01:58:12.672562 14010 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:12.659431 13873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c078b30fc50b475193ae485fbe10e04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.672827 13951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 632bb79cb33a491fa5765cd80357f1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.675406 13798 tablet_service.cc:1468] Processing CreateTablet for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250623 01:58:12.687580 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.681818 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:12.692534 13779 leader_election.cc:343] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:12.692966 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:12.693737 14009 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:12.649298 14003 tablet_bootstrap.cc:654] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.698552 13794 tablet_service.cc:1468] Processing CreateTablet for tablet c078b30fc50b475193ae485fbe10e04c (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250623 01:58:12.718322 13952 tablet_service.cc:1468] Processing CreateTablet for tablet e92cd39500a645e79b368d26acb587ed (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250623 01:58:12.719610 13952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92cd39500a645e79b368d26acb587ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.719369 13797 tablet_service.cc:1468] Processing CreateTablet for tablet 7560a94e265b433dbc427ef13c120669 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250623 01:58:12.698781 13871 tablet_service.cc:1468] Processing CreateTablet for tablet 86045b40aeea4a02993bcd65deecd670 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250623 01:58:12.699388 13803 tablet_service.cc:1468] Processing CreateTablet for tablet 6f13ab3fda6f4c6b9b68d33e7a89c988 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250623 01:58:12.701576 14004 tablet_bootstrap.cc:654] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.661188 13950 tablet_service.cc:1468] Processing CreateTablet for tablet 7560a94e265b433dbc427ef13c120669 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250623 01:58:12.702608 14002 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:12.705404 13878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7560a94e265b433dbc427ef13c120669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.683095 13945 tablet_service.cc:1468] Processing CreateTablet for tablet dfe9e13c26c84532b389e343edf7cef4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250623 01:58:12.712520 13877 tablet_service.cc:1468] Processing CreateTablet for tablet dfe9e13c26c84532b389e343edf7cef4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250623 01:58:12.734140 13877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe9e13c26c84532b389e343edf7cef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.733826 13869 tablet_service.cc:1468] Processing CreateTablet for tablet 632bb79cb33a491fa5765cd80357f1a4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250623 01:58:12.688146 13944 tablet_service.cc:1468] Processing CreateTablet for tablet 829478b3e1dc4a95a122fcd8be3d2c77 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250623 01:58:12.723798 13794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c078b30fc50b475193ae485fbe10e04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.725627 14002 ts_tablet_manager.cc:1397] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.401s user 0.034s sys 0.008s
I20250623 01:58:12.728278 13950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7560a94e265b433dbc427ef13c120669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.737802 14002 raft_consensus.cc:357] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.738516 14002 raft_consensus.cc:383] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.738881 14002 raft_consensus.cc:738] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.738404 13802 tablet_service.cc:1468] Processing CreateTablet for tablet 632bb79cb33a491fa5765cd80357f1a4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250623 01:58:12.739657 14002 consensus_queue.cc:260] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.742182 14002 ts_tablet_manager.cc:1428] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250623 01:58:12.743075 14002 tablet_bootstrap.cc:492] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:12.743741 13803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f13ab3fda6f4c6b9b68d33e7a89c988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.743623 13795 tablet_service.cc:1468] Processing CreateTablet for tablet dfe9e13c26c84532b389e343edf7cef4 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250623 01:58:12.744742 13944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829478b3e1dc4a95a122fcd8be3d2c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.744834 13795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe9e13c26c84532b389e343edf7cef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.719828 13951 tablet_service.cc:1468] Processing CreateTablet for tablet 5bd540ef429946c987f593ab36327b5a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250623 01:58:12.746315 13946 tablet_service.cc:1468] Processing CreateTablet for tablet e20d8c11ac244feeaecb254a76559f76 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250623 01:58:12.747562 13951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd540ef429946c987f593ab36327b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.753891 13945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe9e13c26c84532b389e343edf7cef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.735006 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 632bb79cb33a491fa5765cd80357f1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.681294 13947 tablet_service.cc:1468] Processing CreateTablet for tablet 86045b40aeea4a02993bcd65deecd670 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250623 01:58:12.762077 13870 tablet_service.cc:1468] Processing CreateTablet for tablet 5bd540ef429946c987f593ab36327b5a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250623 01:58:12.762921 13948 tablet_service.cc:1468] Processing CreateTablet for tablet 6ac15d89950f4485a84912743c361f1d (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250623 01:58:12.772800 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86045b40aeea4a02993bcd65deecd670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.773478 13802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 632bb79cb33a491fa5765cd80357f1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.753093 13875 tablet_service.cc:1468] Processing CreateTablet for tablet e92cd39500a645e79b368d26acb587ed (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250623 01:58:12.791805 13949 tablet_service.cc:1468] Processing CreateTablet for tablet f44146f651124d8fba4cc1825f480603 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250623 01:58:12.792245 13800 tablet_service.cc:1468] Processing CreateTablet for tablet 86045b40aeea4a02993bcd65deecd670 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250623 01:58:12.776093 13946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20d8c11ac244feeaecb254a76559f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.793116 13870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd540ef429946c987f593ab36327b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.790153 14009 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:12.797601 14009 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.799521 14009 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:12.801153 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:12.801903 13888 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:12.803623 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:12.803843 13776 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:12.804831 13779 leader_election.cc:343] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:12.806286 14009 raft_consensus.cc:2802] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:12.806689 14009 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:12.806401 13801 tablet_service.cc:1468] Processing CreateTablet for tablet 829478b3e1dc4a95a122fcd8be3d2c77 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250623 01:58:12.807040 14009 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.807675 13801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829478b3e1dc4a95a122fcd8be3d2c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.815129 13877 tablet_service.cc:1468] Processing CreateTablet for tablet f44146f651124d8fba4cc1825f480603 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250623 01:58:12.816426 13877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44146f651124d8fba4cc1825f480603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.792729 13872 tablet_service.cc:1468] Processing CreateTablet for tablet 6ac15d89950f4485a84912743c361f1d (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250623 01:58:12.818053 13872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ac15d89950f4485a84912743c361f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.765610 13876 tablet_service.cc:1468] Processing CreateTablet for tablet e20d8c11ac244feeaecb254a76559f76 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250623 01:58:12.762077 13873 tablet_service.cc:1468] Processing CreateTablet for tablet 829478b3e1dc4a95a122fcd8be3d2c77 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250623 01:58:12.822045 13876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20d8c11ac244feeaecb254a76559f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.823637 13798 tablet_service.cc:1468] Processing CreateTablet for tablet 5bd540ef429946c987f593ab36327b5a (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250623 01:58:12.826555 13950 tablet_service.cc:1468] Processing CreateTablet for tablet bb4438d5ce9e41f3a72f00de14912cff (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250623 01:58:12.827607 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7560a94e265b433dbc427ef13c120669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.828207 13800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86045b40aeea4a02993bcd65deecd670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.825707 13948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ac15d89950f4485a84912743c361f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.829578 13949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44146f651124d8fba4cc1825f480603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.831549 13875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92cd39500a645e79b368d26acb587ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.831986 13947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86045b40aeea4a02993bcd65deecd670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.847095 14009 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.853513 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:12.854094 13888 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.860234 13888 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:12.861364 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd540ef429946c987f593ab36327b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.862967 14009 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:12.864233 14002 tablet_bootstrap.cc:654] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.864804 14004 tablet_bootstrap.cc:492] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:12.864879 13873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829478b3e1dc4a95a122fcd8be3d2c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.865329 14004 ts_tablet_manager.cc:1397] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.318s user 0.012s sys 0.019s
I20250623 01:58:12.868571 14003 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:12.869009 14003 ts_tablet_manager.cc:1397] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.240s user 0.014s sys 0.011s
I20250623 01:58:12.870944 13776 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:12.871598 14003 raft_consensus.cc:357] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.872305 14003 raft_consensus.cc:383] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.872586 14003 raft_consensus.cc:738] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.872843 13796 tablet_service.cc:1468] Processing CreateTablet for tablet e92cd39500a645e79b368d26acb587ed (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250623 01:58:12.874193 13796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92cd39500a645e79b368d26acb587ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.874679 14003 consensus_queue.cc:260] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.876081 14004 raft_consensus.cc:357] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.876405 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:12.876709 14004 raft_consensus.cc:383] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.876933 13962 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.877022 14004 raft_consensus.cc:738] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.875615 13874 tablet_service.cc:1468] Processing CreateTablet for tablet bb4438d5ce9e41f3a72f00de14912cff (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250623 01:58:12.881178 13874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4438d5ce9e41f3a72f00de14912cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.883172 14009 raft_consensus.cc:2802] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:12.890977 14004 consensus_queue.cc:260] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.880311 13950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4438d5ce9e41f3a72f00de14912cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.875541 13803 tablet_service.cc:1468] Processing CreateTablet for tablet e20d8c11ac244feeaecb254a76559f76 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250623 01:58:12.898036 13803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20d8c11ac244feeaecb254a76559f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.898835 14004 ts_tablet_manager.cc:1428] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20250623 01:58:12.899727 14004 tablet_bootstrap.cc:492] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:12.908059 13795 tablet_service.cc:1468] Processing CreateTablet for tablet 6ac15d89950f4485a84912743c361f1d (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250623 01:58:12.909245 13795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ac15d89950f4485a84912743c361f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.923137 13943 tablet_service.cc:1468] Processing CreateTablet for tablet 05203153a0ed43df91fde4fe090579a7 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250623 01:58:12.924764 13870 tablet_service.cc:1468] Processing CreateTablet for tablet 05203153a0ed43df91fde4fe090579a7 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250623 01:58:12.926038 13946 tablet_service.cc:1468] Processing CreateTablet for tablet 2f02354fc186482aa0fc28cab32390c5 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250623 01:58:12.930049 14009 raft_consensus.cc:695] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:12.935734 14009 consensus_queue.cc:237] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:12.938071 13869 tablet_service.cc:1468] Processing CreateTablet for tablet 2f02354fc186482aa0fc28cab32390c5 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250623 01:58:12.939294 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f02354fc186482aa0fc28cab32390c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.939254 13945 tablet_service.cc:1468] Processing CreateTablet for tablet 3df3a5b0b6b347b89175eef714fe6a07 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250623 01:58:12.941011 14003 ts_tablet_manager.cc:1428] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.072s user 0.006s sys 0.000s
I20250623 01:58:12.942178 13943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05203153a0ed43df91fde4fe090579a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.943337 13871 tablet_service.cc:1468] Processing CreateTablet for tablet 7c4f89b1590c439a9b723110eaad17a8 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250623 01:58:12.944491 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4f89b1590c439a9b723110eaad17a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.944522 13794 tablet_service.cc:1468] Processing CreateTablet for tablet 7c4f89b1590c439a9b723110eaad17a8 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250623 01:58:12.945935 13794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4f89b1590c439a9b723110eaad17a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.947360 13952 tablet_service.cc:1468] Processing CreateTablet for tablet 7c4f89b1590c439a9b723110eaad17a8 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250623 01:58:12.947039 13962 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:12.932600 13951 tablet_service.cc:1468] Processing CreateTablet for tablet 28d0bf37b0354956a6d195c5259a3d19 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250623 01:58:12.935755 13797 tablet_service.cc:1468] Processing CreateTablet for tablet f44146f651124d8fba4cc1825f480603 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250623 01:58:12.955338 13799 tablet_service.cc:1468] Processing CreateTablet for tablet 3df3a5b0b6b347b89175eef714fe6a07 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250623 01:58:12.956660 13878 tablet_service.cc:1468] Processing CreateTablet for tablet 3df3a5b0b6b347b89175eef714fe6a07 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250623 01:58:12.957095 13870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05203153a0ed43df91fde4fe090579a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.957870 13945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df3a5b0b6b347b89175eef714fe6a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.957214 14003 tablet_bootstrap.cc:492] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:12.979315 14004 tablet_bootstrap.cc:654] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.979596 14002 tablet_bootstrap.cc:492] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:12.980260 14002 ts_tablet_manager.cc:1397] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.237s user 0.017s sys 0.005s
I20250623 01:58:12.981626 13952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4f89b1590c439a9b723110eaad17a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.983839 14002 raft_consensus.cc:357] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:12.984774 14002 raft_consensus.cc:383] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.985594 13946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f02354fc186482aa0fc28cab32390c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.996460 13951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d0bf37b0354956a6d195c5259a3d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.999118 13872 tablet_service.cc:1468] Processing CreateTablet for tablet 28d0bf37b0354956a6d195c5259a3d19 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250623 01:58:13.000329 13872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d0bf37b0354956a6d195c5259a3d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.985131 14002 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.005118 13796 tablet_service.cc:1468] Processing CreateTablet for tablet 2f02354fc186482aa0fc28cab32390c5 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250623 01:58:13.006312 13796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f02354fc186482aa0fc28cab32390c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:13.009663 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44146f651124d8fba4cc1825f480603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:13.012019 14003 tablet_bootstrap.cc:654] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.013561 14002 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:13.016213 14002 ts_tablet_manager.cc:1428] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20250623 01:58:13.016898 13878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df3a5b0b6b347b89175eef714fe6a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:13.018122 13799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df3a5b0b6b347b89175eef714fe6a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.950991 13803 tablet_service.cc:1468] Processing CreateTablet for tablet bb4438d5ce9e41f3a72f00de14912cff (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250623 01:58:13.017182 14002 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.024736 13800 tablet_service.cc:1468] Processing CreateTablet for tablet 05203153a0ed43df91fde4fe090579a7 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250623 01:58:13.037560 13801 tablet_service.cc:1468] Processing CreateTablet for tablet 28d0bf37b0354956a6d195c5259a3d19 (DEFAULT_TABLE table=test-workload [id=865003f5f3954d04903f64a7082491b7]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250623 01:58:13.050138 13710 catalog_manager.cc:5582] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.053685 13803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4438d5ce9e41f3a72f00de14912cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:13.054893 13800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05203153a0ed43df91fde4fe090579a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:13.062148 14002 tablet_bootstrap.cc:654] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.073150 13801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d0bf37b0354956a6d195c5259a3d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:13.086452 14004 tablet_bootstrap.cc:492] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.086946 14004 ts_tablet_manager.cc:1397] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.187s user 0.022s sys 0.000s
I20250623 01:58:13.089550 14004 raft_consensus.cc:357] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.090373 14004 raft_consensus.cc:383] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.090766 14004 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.091758 14004 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.094528 14004 ts_tablet_manager.cc:1428] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250623 01:58:13.095539 14004 tablet_bootstrap.cc:492] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.109972 14003 tablet_bootstrap.cc:492] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.110396 14003 ts_tablet_manager.cc:1397] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.153s user 0.014s sys 0.005s
I20250623 01:58:13.114629 14003 raft_consensus.cc:357] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.115255 14003 raft_consensus.cc:383] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.115504 14003 raft_consensus.cc:738] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.116263 14003 consensus_queue.cc:260] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.121289 14003 ts_tablet_manager.cc:1428] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250623 01:58:13.122256 14003 tablet_bootstrap.cc:492] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.125211 14009 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.125830 14009 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:13.128964 14009 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:13.129896 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.130234 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:13.131745 14004 tablet_bootstrap.cc:654] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:13.131978 13776 leader_election.cc:343] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.132630 13962 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:13.133177 14003 tablet_bootstrap.cc:654] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.133930 13779 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:13.135006 14009 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.135322 14009 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.135684 14009 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.142453 14002 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.142938 14002 ts_tablet_manager.cc:1397] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.126s user 0.026s sys 0.022s
I20250623 01:58:13.146193 14002 raft_consensus.cc:357] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.146881 14002 raft_consensus.cc:383] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.147118 14002 raft_consensus.cc:738] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.147305 14003 tablet_bootstrap.cc:492] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.147789 14004 tablet_bootstrap.cc:492] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.147727 14003 ts_tablet_manager.cc:1397] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.004s
I20250623 01:58:13.147703 14002 consensus_queue.cc:260] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.148319 14004 ts_tablet_manager.cc:1397] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.009s
I20250623 01:58:13.148497 14009 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:13.150017 14002 ts_tablet_manager.cc:1428] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20250623 01:58:13.150921 14002 tablet_bootstrap.cc:492] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.150557 14003 raft_consensus.cc:357] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.150682 14004 raft_consensus.cc:357] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.151237 14003 raft_consensus.cc:383] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.151299 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.151780 14003 raft_consensus.cc:738] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.151996 13962 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.152947 14003 consensus_queue.cc:260] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.155679 14003 ts_tablet_manager.cc:1428] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250623 01:58:13.156695 14003 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.151269 14004 raft_consensus.cc:383] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.157560 14004 raft_consensus.cc:738] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.158010 14002 tablet_bootstrap.cc:654] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.158318 14004 consensus_queue.cc:260] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.159664 13962 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:13.160964 14004 ts_tablet_manager.cc:1428] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250623 01:58:13.160871 13779 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:13.162027 14004 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.162207 14022 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.162755 14003 tablet_bootstrap.cc:654] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.170074 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:13.171789 13776 leader_election.cc:343] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.173139 14009 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:13.174772 14004 tablet_bootstrap.cc:654] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:13.176887 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:13.178145 14022 raft_consensus.cc:695] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:13.179262 14022 consensus_queue.cc:237] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:13.188403 14002 tablet_bootstrap.cc:492] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.189070 14002 ts_tablet_manager.cc:1397] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.024s
I20250623 01:58:13.192616 14002 raft_consensus.cc:357] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.193650 14002 raft_consensus.cc:383] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.194048 14002 raft_consensus.cc:738] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.194818 14002 consensus_queue.cc:260] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.197151 14002 ts_tablet_manager.cc:1428] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250623 01:58:13.198231 14002 tablet_bootstrap.cc:492] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.200215 14004 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.200695 14004 ts_tablet_manager.cc:1397] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.007s
I20250623 01:58:13.203244 14003 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.203716 14003 ts_tablet_manager.cc:1397] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.008s
I20250623 01:58:13.203338 14004 raft_consensus.cc:357] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.204079 14004 raft_consensus.cc:383] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.204352 14004 raft_consensus.cc:738] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.204454 14002 tablet_bootstrap.cc:654] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.203840 13710 catalog_manager.cc:5582] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.205236 14004 consensus_queue.cc:260] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.206435 14003 raft_consensus.cc:357] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.207104 14003 raft_consensus.cc:383] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.207429 14003 raft_consensus.cc:738] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.207695 14004 ts_tablet_manager.cc:1428] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20250623 01:58:13.208710 14004 tablet_bootstrap.cc:492] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.208422 14003 consensus_queue.cc:260] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.211694 14003 ts_tablet_manager.cc:1428] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250623 01:58:13.212941 14003 tablet_bootstrap.cc:492] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.214745 14004 tablet_bootstrap.cc:654] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.220610 14003 tablet_bootstrap.cc:654] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.225656 14004 tablet_bootstrap.cc:492] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.226101 14004 ts_tablet_manager.cc:1397] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20250623 01:58:13.231024 14002 tablet_bootstrap.cc:492] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.231535 14002 ts_tablet_manager.cc:1397] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.014s
I20250623 01:58:13.231582 14003 tablet_bootstrap.cc:492] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.232235 14003 ts_tablet_manager.cc:1397] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
I20250623 01:58:13.231904 14004 raft_consensus.cc:357] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.232661 14004 raft_consensus.cc:383] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.232952 14004 raft_consensus.cc:738] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.233711 14004 consensus_queue.cc:260] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.234772 14002 raft_consensus.cc:357] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.235502 14002 raft_consensus.cc:383] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.235836 14002 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.236640 14003 raft_consensus.cc:357] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.236550 14002 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.237325 14003 raft_consensus.cc:383] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.237622 14003 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.238343 14003 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.238999 14002 ts_tablet_manager.cc:1428] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250623 01:58:13.239877 14002 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.240783 14003 ts_tablet_manager.cc:1428] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250623 01:58:13.241744 14003 tablet_bootstrap.cc:492] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.235997 14004 ts_tablet_manager.cc:1428] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20250623 01:58:13.246137 14004 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.253458 14002 tablet_bootstrap.cc:654] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.253952 14004 tablet_bootstrap.cc:654] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.253458 14003 tablet_bootstrap.cc:654] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.258888 14004 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.259222 14004 ts_tablet_manager.cc:1397] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20250623 01:58:13.260380 14003 tablet_bootstrap.cc:492] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.260812 14003 ts_tablet_manager.cc:1397] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.006s
I20250623 01:58:13.261670 14002 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.262094 14002 ts_tablet_manager.cc:1397] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.003s
I20250623 01:58:13.261718 14004 raft_consensus.cc:357] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.262507 14004 raft_consensus.cc:383] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.262804 14004 raft_consensus.cc:738] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.263217 14003 raft_consensus.cc:357] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.263828 14003 raft_consensus.cc:383] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.263458 14004 consensus_queue.cc:260] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.263970 14013 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.264088 14003 raft_consensus.cc:738] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.264302 14002 raft_consensus.cc:357] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.264556 14013 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.264995 14002 raft_consensus.cc:383] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.265424 14002 raft_consensus.cc:738] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.264991 14003 consensus_queue.cc:260] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.266047 14002 consensus_queue.cc:260] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.267763 14003 ts_tablet_manager.cc:1428] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250623 01:58:13.268196 14002 ts_tablet_manager.cc:1428] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20250623 01:58:13.268790 14003 tablet_bootstrap.cc:492] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.271374 14002 tablet_bootstrap.cc:492] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.276297 14013 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.276335 14003 tablet_bootstrap.cc:654] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.277873 14002 tablet_bootstrap.cc:654] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.278354 14004 ts_tablet_manager.cc:1428] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.019s user 0.010s sys 0.009s
I20250623 01:58:13.283113 14002 tablet_bootstrap.cc:492] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.283578 14002 ts_tablet_manager.cc:1397] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250623 01:58:13.285940 14002 raft_consensus.cc:357] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.286552 14002 raft_consensus.cc:383] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.286664 14004 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.286821 14002 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.287447 14002 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.292970 14002 ts_tablet_manager.cc:1428] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20250623 01:58:13.293064 14004 tablet_bootstrap.cc:654] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.293867 14002 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.297539 14013 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.299294 14002 tablet_bootstrap.cc:654] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.299345 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:13.298717 14013 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.302155 13813 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.302851 14013 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.304355 14003 tablet_bootstrap.cc:492] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.304719 14003 ts_tablet_manager.cc:1397] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.000s
I20250623 01:58:13.303525 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:13.305478 13812 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.305444 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.306676 13852 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.307735 14044 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.307857 14013 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.308121 14044 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.308907 14044 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.309208 14003 raft_consensus.cc:357] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.309801 14003 raft_consensus.cc:383] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.309998 14003 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.308521 14013 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.313562 14013 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.314749 14003 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.315721 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.315778 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.316865 13962 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.316937 13961 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.316937 14003 ts_tablet_manager.cc:1428] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250623 01:58:13.318684 14003 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.319427 14044 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.321269 14044 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.322434 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:13.323102 13812 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.323274 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.323834 13961 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.325176 14003 tablet_bootstrap.cc:654] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.326159 14004 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.326488 14004 ts_tablet_manager.cc:1397] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.015s
I20250623 01:58:13.331553 13812 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.332744 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.333570 14044 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.334733 13961 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.338176 14044 raft_consensus.cc:695] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:13.338317 14004 raft_consensus.cc:357] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.338994 14004 raft_consensus.cc:383] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.339282 14004 raft_consensus.cc:738] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.339568 14002 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.339087 14044 consensus_queue.cc:237] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.339896 14004 consensus_queue.cc:260] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.341621 14002 ts_tablet_manager.cc:1397] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.008s
I20250623 01:58:13.345971 14013 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.345897 14002 raft_consensus.cc:357] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.347136 14002 raft_consensus.cc:383] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.347554 14002 raft_consensus.cc:738] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.348363 14002 consensus_queue.cc:260] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.348867 14022 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.342378 14053 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.349399 14022 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.349812 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.350947 13961 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.349594 14053 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.351356 14022 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.352437 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.352931 14053 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:13.353564 13779 leader_election.cc:343] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.354058 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:13.354470 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.354635 13813 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.352857 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:13.355052 13962 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.355347 14002 ts_tablet_manager.cc:1428] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.012s user 0.002s sys 0.006s
I20250623 01:58:13.355509 13888 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:13.356298 14002 tablet_bootstrap.cc:492] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.356865 13854 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.357646 14053 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.358008 14053 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.358315 14053 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.359021 13961 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.359705 14013 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.342144 14004 ts_tablet_manager.cc:1428] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250623 01:58:13.360816 14004 tablet_bootstrap.cc:492] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.360213 13776 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:13.360132 13854 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.361477 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:13.361763 14033 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.362059 13812 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.362177 14033 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.362569 14033 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.363307 14002 tablet_bootstrap.cc:654] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.369904 14013 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.370473 14004 tablet_bootstrap.cc:654] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.371312 13812 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.375883 14033 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.380594 14057 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.381420 14013 raft_consensus.cc:695] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:13.381115 14057 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.382274 14013 consensus_queue.cc:237] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.382284 14003 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.384630 14057 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.385577 14004 tablet_bootstrap.cc:492] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.385993 14004 ts_tablet_manager.cc:1397] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.006s
I20250623 01:58:13.384728 14003 ts_tablet_manager.cc:1397] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.004s
I20250623 01:58:13.384729 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.387569 13961 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.388640 13854 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.385378 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:13.390954 14004 raft_consensus.cc:357] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.391603 14004 raft_consensus.cc:383] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 01:58:13.391813 13852 leader_election.cc:343] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.392387 14044 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.392751 14044 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.389187 14003 raft_consensus.cc:357] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.393110 14044 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.393258 14003 raft_consensus.cc:383] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.393566 14003 raft_consensus.cc:738] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.394187 14003 consensus_queue.cc:260] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.397297 13711 catalog_manager.cc:5582] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.399706 14044 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.391877 14004 raft_consensus.cc:738] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.401834 14004 consensus_queue.cc:260] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.403805 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:13.403553 14053 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.404351 13888 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.404633 14004 ts_tablet_manager.cc:1428] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250623 01:58:13.405543 14004 tablet_bootstrap.cc:492] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.407663 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.408360 13961 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.414028 14004 tablet_bootstrap.cc:654] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.416404 14033 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.417702 14003 ts_tablet_manager.cc:1428] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20250623 01:58:13.419023 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:13.419734 13813 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.419983 13961 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:13.420277 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:13.421531 13852 leader_election.cc:343] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.421741 14044 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.422406 14003 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.419983 13888 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:13.425895 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.426649 13962 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.426790 13776 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.440609 14003 tablet_bootstrap.cc:654] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.443275 13711 catalog_manager.cc:5582] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.449539 14010 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.450155 14010 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.452836 14010 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:13.454244 14002 tablet_bootstrap.cc:492] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.445081 14033 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.454694 14002 ts_tablet_manager.cc:1397] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.099s user 0.022s sys 0.060s
I20250623 01:58:13.455482 14033 raft_consensus.cc:695] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:13.458081 14033 consensus_queue.cc:237] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.458633 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:13.463804 13812 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:13.465070 13926 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:13.465885 14053 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.465885 14022 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:13.466607 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.467594 13961 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.467881 13813 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.469201 13852 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.460080 13962 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.461925 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:13.470459 14044 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.470508 13888 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:13.470959 13854 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:13.470996 14044 raft_consensus.cc:695] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:13.472636 14013 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.472374 14044 consensus_queue.cc:237] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.473125 14013 raft_consensus.cc:695] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:13.462147 14002 raft_consensus.cc:357] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.473711 14002 raft_consensus.cc:383] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.473940 14010 raft_consensus.cc:2802] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.474754 14002 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.474937 14010 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.475850 14010 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.477859 14013 consensus_queue.cc:237] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.483881 13961 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.484504 14010 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.475795 14002 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.488821 14010 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:13.492193 14002 ts_tablet_manager.cc:1428] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.037s user 0.007s sys 0.001s
I20250623 01:58:13.493000 14002 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.492956 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:13.493505 14003 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.493963 14003 ts_tablet_manager.cc:1397] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.072s user 0.018s sys 0.019s
I20250623 01:58:13.494210 13888 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.494930 14004 tablet_bootstrap.cc:492] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.495450 14004 ts_tablet_manager.cc:1397] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.090s user 0.028s sys 0.014s
I20250623 01:58:13.489420 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:13.499688 13813 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.500236 14003 raft_consensus.cc:357] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.500936 14003 raft_consensus.cc:383] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.501195 14003 raft_consensus.cc:738] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.501911 14003 consensus_queue.cc:260] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.506709 14003 ts_tablet_manager.cc:1428] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250623 01:58:13.509392 13888 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:13.510182 14004 raft_consensus.cc:357] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:13.510825 14004 raft_consensus.cc:383] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.510341 13711 catalog_manager.cc:5582] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.511060 14004 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.511734 14004 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:13.514165 14004 ts_tablet_manager.cc:1428] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
I20250623 01:58:13.514990 14004 tablet_bootstrap.cc:492] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.517288 13925 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.518368 14010 raft_consensus.cc:2802] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.518510 13813 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:13.532300 14003 tablet_bootstrap.cc:492] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.537745 14002 tablet_bootstrap.cc:654] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.536550 14033 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.542361 14033 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:13.543292 13988 tablet.cc:2378] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:58:13.546619 14033 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.550364 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.551204 13961 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:13.552289 13779 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:13.549238 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:13.553606 14033 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.554085 14033 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.554618 14033 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.549887 14010 raft_consensus.cc:695] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:58:13.555684 13776 leader_election.cc:343] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.558441 14010 consensus_queue.cc:237] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.556809 13711 catalog_manager.cc:5582] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.572038 14022 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:13.575490 14003 tablet_bootstrap.cc:654] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.571463 13711 catalog_manager.cc:5582] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.577507 14009 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.578099 14009 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.578677 14033 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.582078 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:13.584873 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.579921 14009 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.585492 13962 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:13.585794 14013 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:13.591368 13776 leader_election.cc:343] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.594872 14013 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:13.595597 13779 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:13.596714 14009 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.597074 14009 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.597479 14013 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:13.597448 14009 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.599165 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:13.600638 14004 tablet_bootstrap.cc:654] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.606400 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.607430 13962 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:13.610842 13711 catalog_manager.cc:5582] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:13.612349 14002 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.612818 14002 ts_tablet_manager.cc:1397] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.120s user 0.013s sys 0.005s
I20250623 01:58:13.615710 14002 raft_consensus.cc:357] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.616346 14002 raft_consensus.cc:383] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.616614 14002 raft_consensus.cc:738] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.617201 14002 consensus_queue.cc:260] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.617879 13854 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:13.619115 14044 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.621498 14013 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.622079 14013 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.624543 14013 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.627094 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:13.627909 13813 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.628157 14002 ts_tablet_manager.cc:1428] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250623 01:58:13.629499 13852 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.631505 14044 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:13.632097 14044 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.632418 14044 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.632750 14044 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.633224 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.634198 13962 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:13.635317 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:13.636420 13776 leader_election.cc:343] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.615710 14009 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.638314 14033 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.639277 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:13.639276 14002 tablet_bootstrap.cc:492] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
W20250623 01:58:13.640422 13776 leader_election.cc:343] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.640337 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.642112 13962 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.647990 14002 tablet_bootstrap.cc:654] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.648962 13962 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:13.650203 13779 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:13.651393 14033 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.651998 14033 raft_consensus.cc:695] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:13.653048 14033 consensus_queue.cc:237] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.664186 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.663964 13709 catalog_manager.cc:5582] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.665212 14009 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.664891 13961 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.671146 14044 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.672951 14004 tablet_bootstrap.cc:492] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.673498 14004 ts_tablet_manager.cc:1397] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.159s user 0.025s sys 0.044s
I20250623 01:58:13.673872 14003 tablet_bootstrap.cc:492] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.674304 14003 ts_tablet_manager.cc:1397] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.143s user 0.009s sys 0.002s
I20250623 01:58:13.676268 14003 raft_consensus.cc:357] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.676734 14003 raft_consensus.cc:383] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.676308 14004 raft_consensus.cc:357] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.676944 14003 raft_consensus.cc:738] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.677233 14004 raft_consensus.cc:383] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.677598 14004 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.677773 14003 consensus_queue.cc:260] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.678194 14004 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.678259 14044 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.679013 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:13.679558 13813 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.680267 14004 ts_tablet_manager.cc:1428] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250623 01:58:13.681713 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.682255 13962 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.685109 14004 tablet_bootstrap.cc:492] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.687348 14009 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:13.689666 13962 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.690769 13854 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.691550 14003 ts_tablet_manager.cc:1428] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20250623 01:58:13.692641 14003 tablet_bootstrap.cc:492] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.691601 14044 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.693917 14044 raft_consensus.cc:695] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:13.695041 14004 tablet_bootstrap.cc:654] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.694742 14044 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.698424 14002 tablet_bootstrap.cc:492] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.698848 14002 ts_tablet_manager.cc:1397] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.013s
I20250623 01:58:13.700390 14003 tablet_bootstrap.cc:654] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.702770 13961 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:13.701646 14002 raft_consensus.cc:357] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.704056 13779 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:13.704768 14002 raft_consensus.cc:383] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.705075 14002 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.705767 14002 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.706418 14033 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.703402 13813 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:13.722501 14002 ts_tablet_manager.cc:1428] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250623 01:58:13.723441 14002 tablet_bootstrap.cc:492] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.722963 14009 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:13.736016 14033 raft_consensus.cc:695] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:13.737068 14033 consensus_queue.cc:237] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.748871 14002 tablet_bootstrap.cc:654] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.755010 14004 tablet_bootstrap.cc:492] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.755503 14004 ts_tablet_manager.cc:1397] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.026s
I20250623 01:58:13.755843 13709 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.757462 14004 raft_consensus.cc:357] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.758714 14004 raft_consensus.cc:383] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.758975 14004 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.759598 14004 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.761214 14004 ts_tablet_manager.cc:1428] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250623 01:58:13.762086 14004 tablet_bootstrap.cc:492] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.767992 14004 tablet_bootstrap.cc:654] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.770612 14003 tablet_bootstrap.cc:492] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.771138 14003 ts_tablet_manager.cc:1397] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.038s
I20250623 01:58:13.774317 14003 raft_consensus.cc:357] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.775134 14003 raft_consensus.cc:383] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.775502 14003 raft_consensus.cc:738] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.776250 14003 consensus_queue.cc:260] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.778250 14003 ts_tablet_manager.cc:1428] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250623 01:58:13.778465 14004 tablet_bootstrap.cc:492] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.778853 14004 ts_tablet_manager.cc:1397] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20250623 01:58:13.779073 14003 tablet_bootstrap.cc:492] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.781081 14004 raft_consensus.cc:357] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.781749 14004 raft_consensus.cc:383] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.782074 14004 raft_consensus.cc:738] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.782847 14004 consensus_queue.cc:260] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.783788 14002 tablet_bootstrap.cc:492] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.784365 14002 ts_tablet_manager.cc:1397] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.000s
I20250623 01:58:13.785344 14004 ts_tablet_manager.cc:1428] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250623 01:58:13.786303 14004 tablet_bootstrap.cc:492] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.787204 14002 raft_consensus.cc:357] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.787834 14002 raft_consensus.cc:383] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.788084 14002 raft_consensus.cc:738] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.788762 14002 consensus_queue.cc:260] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.789844 14003 tablet_bootstrap.cc:654] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.791044 14002 ts_tablet_manager.cc:1428] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250623 01:58:13.791899 14002 tablet_bootstrap.cc:492] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.792609 14004 tablet_bootstrap.cc:654] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.794754 14003 tablet_bootstrap.cc:492] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.795174 14003 ts_tablet_manager.cc:1397] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20250623 01:58:13.796389 13710 catalog_manager.cc:5582] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:13.797911 14033 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.798292 14002 tablet_bootstrap.cc:654] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.798520 14033 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.799921 14003 raft_consensus.cc:357] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.800794 14003 raft_consensus.cc:383] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.801018 14033 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.801134 14003 raft_consensus.cc:738] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.802157 14033 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.803517 14033 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.804804 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.805685 14033 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.802556 14003 consensus_queue.cc:260] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.809959 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.810151 13960 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:13.810751 14003 ts_tablet_manager.cc:1428] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
W20250623 01:58:13.811297 13779 leader_election.cc:343] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.811621 14003 tablet_bootstrap.cc:492] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.816721 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:13.818859 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:13.819464 13887 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
W20250623 01:58:13.820600 13776 leader_election.cc:343] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.820785 13779 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:13.821169 13776 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:13.821599 14033 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.823040 14033 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.823274 14082 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:13.823426 14033 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.826996 14003 tablet_bootstrap.cc:654] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.830624 14004 tablet_bootstrap.cc:492] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:13.831070 14004 ts_tablet_manager.cc:1397] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.007s
I20250623 01:58:13.831158 14033 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.835976 14002 tablet_bootstrap.cc:492] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:13.836406 14002 ts_tablet_manager.cc:1397] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.000s
I20250623 01:58:13.838514 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:13.839011 14003 tablet_bootstrap.cc:492] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:13.833773 14004 raft_consensus.cc:357] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.839052 13887 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.839567 14004 raft_consensus.cc:383] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.839634 14003 ts_tablet_manager.cc:1397] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.004s
I20250623 01:58:13.839900 14004 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.840600 14004 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.842710 14004 ts_tablet_manager.cc:1428] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250623 01:58:13.838991 14002 raft_consensus.cc:357] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.843488 14002 raft_consensus.cc:383] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.843750 14002 raft_consensus.cc:738] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.843675 14003 raft_consensus.cc:357] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.844408 14003 raft_consensus.cc:383] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:13.844798 14003 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:13.844434 14002 consensus_queue.cc:260] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.847172 14002 ts_tablet_manager.cc:1428] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250623 01:58:13.847965 14004 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:13.848145 13887 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:13.849290 13776 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:13.849494 14003 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.850036 14082 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:13.850481 14082 raft_consensus.cc:695] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:13.850867 14033 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.851325 14082 consensus_queue.cc:237] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.853575 14004 tablet_bootstrap.cc:654] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.855252 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:13.855827 14002 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:13.855809 13960 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.859972 14044 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:13.861819 14003 ts_tablet_manager.cc:1428] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250623 01:58:13.864697 14002 tablet_bootstrap.cc:654] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:13.867945 13852 consensus_peers.cc:487] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:13.868996 14013 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:13.870469 14003 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:13.873631 14013 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:13.883605 14003 tablet_bootstrap.cc:654] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:13.907388 14013 consensus_queue.cc:1035] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:13.919593 14013 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:13.960487 14044 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:13.961016 14044 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.963092 14044 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:13.964295 14013 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:13.966228 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:13.966423 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:13.966943 13958 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
W20250623 01:58:13.967643 13852 leader_election.cc:343] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:13.968195 13854 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:13.965406 14044 consensus_queue.cc:1035] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 01:58:13.970278 14057 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:13.970659 14057 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:13.970985 14057 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:13.978576 13960 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:13.979544 14057 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:13.985064 14067 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:13.997871 14022 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:14.013059 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:14.026551 14081 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.027068 14081 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.028962 14081 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.031980 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:14.032617 13957 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
W20250623 01:58:14.030877 13852 leader_election.cc:343] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.037056 14013 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.036936 13779 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:14.037127 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.035583 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:14.037683 14013 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.038123 14081 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:14.038580 14081 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:58:14.038647 13776 leader_election.cc:343] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.039043 14081 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.039477 14013 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.038113 13957 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.036235 14057 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.036922 14082 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.064208 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:14.065244 13956 raft_consensus.cc:2391] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:14.062497 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:14.076965 14082 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.086865 14013 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.098203 14022 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:14.101734 14013 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.135782 14022 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:14.176025 13852 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:14.188670 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:14.190521 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.192761 14002 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:14.192770 14004 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:14.193352 14004 ts_tablet_manager.cc:1397] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.346s user 0.065s sys 0.140s
I20250623 01:58:14.191213 13887 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:14.193334 14002 ts_tablet_manager.cc:1397] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.338s user 0.076s sys 0.089s
I20250623 01:58:14.192879 14013 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.200348 14003 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:14.200714 14003 ts_tablet_manager.cc:1397] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.330s user 0.069s sys 0.072s
I20250623 01:58:14.198390 14010 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.200011 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:14.201534 14092 log.cc:927] Time spent T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20250623 01:58:14.201902 14010 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.172915 14081 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.205209 14108 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.194747 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:14.206017 14004 raft_consensus.cc:357] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.210238 14004 raft_consensus.cc:383] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.206414 14067 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:14.207126 14010 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:14.210525 14004 raft_consensus.cc:738] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.195698 14057 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250623 01:58:14.211091 14082 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.210418 14022 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:14.235184 14022 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:14.235569 14022 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:58:14.229504 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:14.212865 14004 consensus_queue.cc:260] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.219870 14002 raft_consensus.cc:357] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.219116 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:58:14.216148 14100 log.cc:927] Time spent T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20250623 01:58:14.220188 14003 raft_consensus.cc:357] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.232873 14009 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:14.214048 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:14.235863 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.241194 13887 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:14.242803 13925 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:14.244941 13887 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.245514 14004 ts_tablet_manager.cc:1428] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20250623 01:58:14.213943 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:14.246552 14004 tablet_bootstrap.cc:492] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:14.248661 14022 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.249734 13813 raft_consensus.cc:2391] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:14.239966 14003 raft_consensus.cc:383] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.250627 14003 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.238576 14002 raft_consensus.cc:383] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.251609 14002 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.252097 14002 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.251145 14003 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.243571 14010 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:14.254878 14010 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:14.255221 14010 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.258672 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:14.261469 13776 leader_election.cc:343] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.264317 14022 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.263558 13710 catalog_manager.cc:5582] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:14.264995 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:14.250480 14108 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:14.263872 14010 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.267117 13888 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.267258 14104 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:14.268318 14057 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:14.274012 14002 ts_tablet_manager.cc:1428] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.079s user 0.002s sys 0.007s
I20250623 01:58:14.274093 14003 ts_tablet_manager.cc:1428] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.073s user 0.004s sys 0.006s
I20250623 01:58:14.277477 14003 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:14.273881 14081 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.288120 14002 tablet_bootstrap.cc:492] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:14.289463 13962 raft_consensus.cc:2391] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:14.281530 13957 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:14.278808 14082 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250623 01:58:14.310293 13854 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:14.281713 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:14.319312 14106 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:14.319844 14106 raft_consensus.cc:695] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:14.320675 14106 consensus_queue.cc:237] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.304888 14010 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:14.325006 14009 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:14.326982 13888 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:14.330601 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:14.329048 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:14.357720 13885 raft_consensus.cc:2391] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:14.371361 13954 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.465386 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:14.466465 14082 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:14.467013 14082 raft_consensus.cc:695] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:14.467954 14082 consensus_queue.cc:237] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.477433 14002 tablet_bootstrap.cc:654] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.481863 14104 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.487426 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:14.498996 14004 tablet_bootstrap.cc:654] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.487422 14104 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.530504 14104 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.532366 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:14.538993 13813 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:14.567003 14003 tablet_bootstrap.cc:654] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.554493 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:14.553443 14081 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20250623 01:58:14.557054 13852 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:14.570777 14002 tablet_bootstrap.cc:492] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:14.571296 14002 ts_tablet_manager.cc:1397] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.285s user 0.015s sys 0.023s
I20250623 01:58:14.570716 14106 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:14.569031 13926 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:14.572161 14106 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:14.572535 14106 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.573989 14111 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250623 01:58:14.577174 13854 leader_election.cc:343] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.573290 14033 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250623 01:58:14.586135 13954 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:14.587620 14106 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.590984 14106 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.591192 13779 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:14.592563 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:14.592602 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:14.593704 14009 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:14.594425 14009 raft_consensus.cc:695] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
W20250623 01:58:14.597743 13854 leader_election.cc:343] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.593976 13811 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.595229 14009 consensus_queue.cc:237] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.611112 14002 raft_consensus.cc:357] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.619282 14002 raft_consensus.cc:383] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.619648 14002 raft_consensus.cc:738] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.620374 14002 consensus_queue.cc:260] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.699311 14002 ts_tablet_manager.cc:1428] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.128s user 0.004s sys 0.004s
I20250623 01:58:14.713291 14010 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.713965 14010 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.716557 14010 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:14.716919 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:14.717700 13813 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:14.720319 13926 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:14.722450 14010 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:14.722993 14010 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:14.722859 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.724116 14010 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:58:14.725215 13925 leader_election.cc:343] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.733016 14003 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:14.733456 14003 ts_tablet_manager.cc:1397] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.458s user 0.017s sys 0.019s
I20250623 01:58:14.735747 14003 raft_consensus.cc:357] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.736362 14003 raft_consensus.cc:383] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.736598 14003 raft_consensus.cc:738] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.736882 14002 tablet_bootstrap.cc:492] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:14.737123 14003 consensus_queue.cc:260] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.740170 14003 ts_tablet_manager.cc:1428] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250623 01:58:14.740919 14033 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.741524 14033 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.744218 14033 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.745033 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:14.745885 13961 raft_consensus.cc:2391] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:14.741065 14003 tablet_bootstrap.cc:492] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:14.741750 13811 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:14.751161 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.750867 13852 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:14.752344 14104 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:14.752817 14104 raft_consensus.cc:695] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:14.753613 14104 consensus_queue.cc:237] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.756027 13776 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:14.757122 14081 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.763326 14003 tablet_bootstrap.cc:654] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.733448 14004 tablet_bootstrap.cc:492] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:14.766330 14004 ts_tablet_manager.cc:1397] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.520s user 0.033s sys 0.028s
I20250623 01:58:14.766417 13710 catalog_manager.cc:5582] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:14.769289 14004 raft_consensus.cc:357] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.770157 14004 raft_consensus.cc:383] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.770475 14004 raft_consensus.cc:738] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.764436 14081 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:14.767163 14010 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.776324 14004 consensus_queue.cc:260] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.778661 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:14.779788 13925 leader_election.cc:343] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.780526 14010 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:14.780804 14004 ts_tablet_manager.cc:1428] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20250623 01:58:14.781668 14004 tablet_bootstrap.cc:492] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:14.785223 14003 tablet_bootstrap.cc:492] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:14.785704 14003 ts_tablet_manager.cc:1397] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.015s
I20250623 01:58:14.787344 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:14.787848 14004 tablet_bootstrap.cc:654] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.787930 13811 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.787940 14003 raft_consensus.cc:357] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.788785 14003 raft_consensus.cc:383] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.789057 14003 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.789741 14003 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.792016 14003 ts_tablet_manager.cc:1428] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250623 01:58:14.793118 14003 tablet_bootstrap.cc:492] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:14.794739 13811 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:14.800943 13926 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:14.802030 14010 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:14.802716 14003 tablet_bootstrap.cc:654] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.803001 14010 raft_consensus.cc:695] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:14.803776 14010 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.807945 14004 tablet_bootstrap.cc:492] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:14.808367 14004 ts_tablet_manager.cc:1397] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.004s
I20250623 01:58:14.810936 14004 raft_consensus.cc:357] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.811568 14004 raft_consensus.cc:383] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.811816 14002 tablet_bootstrap.cc:654] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.813470 14003 tablet_bootstrap.cc:492] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:14.813866 14003 ts_tablet_manager.cc:1397] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20250623 01:58:14.816049 14003 raft_consensus.cc:357] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.816644 14003 raft_consensus.cc:383] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.816901 14003 raft_consensus.cc:738] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.817483 14003 consensus_queue.cc:260] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.819839 14003 ts_tablet_manager.cc:1428] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250623 01:58:14.820775 14003 tablet_bootstrap.cc:492] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:14.811816 14004 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.822042 14004 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.823561 14004 ts_tablet_manager.cc:1428] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250623 01:58:14.823661 14002 tablet_bootstrap.cc:492] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:14.824088 14002 ts_tablet_manager.cc:1397] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.087s user 0.014s sys 0.002s
I20250623 01:58:14.824173 14004 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:14.823952 13708 catalog_manager.cc:5582] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:14.826333 14002 raft_consensus.cc:357] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.826241 13708 catalog_manager.cc:5582] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:14.826920 14002 raft_consensus.cc:383] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.827343 14002 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.828203 14002 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.829989 14004 tablet_bootstrap.cc:654] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.830466 14003 tablet_bootstrap.cc:654] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.830814 14002 ts_tablet_manager.cc:1428] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250623 01:58:14.831771 14002 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:14.840157 14002 tablet_bootstrap.cc:654] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.846778 14106 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:14.853035 13709 catalog_manager.cc:5582] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:14.865697 14004 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:14.866231 14004 ts_tablet_manager.cc:1397] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.042s user 0.018s sys 0.016s
I20250623 01:58:14.880103 13710 catalog_manager.cc:5582] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:14.878295 14114 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 01:58:14.887497 14004 raft_consensus.cc:357] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.888095 14004 raft_consensus.cc:383] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.888314 14004 raft_consensus.cc:738] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.888914 14004 consensus_queue.cc:260] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.891608 14004 ts_tablet_manager.cc:1428] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250623 01:58:14.892704 14004 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:14.893196 14081 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.893703 14081 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.901654 14009 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:14.903944 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:14.905550 13961 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:14.904153 14081 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.906949 13779 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:14.909571 14009 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:14.909910 14009 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:14.906831 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.910231 14009 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.911803 14004 tablet_bootstrap.cc:654] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.912261 13885 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:14.917598 14003 tablet_bootstrap.cc:492] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:14.919617 14003 ts_tablet_manager.cc:1397] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.099s user 0.039s sys 0.015s
I20250623 01:58:14.921272 14002 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:14.921743 14002 ts_tablet_manager.cc:1397] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.090s user 0.026s sys 0.006s
I20250623 01:58:14.922394 14003 raft_consensus.cc:357] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.923082 14003 raft_consensus.cc:383] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.923333 14003 raft_consensus.cc:738] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.923960 14003 consensus_queue.cc:260] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.924124 14002 raft_consensus.cc:357] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.924855 14002 raft_consensus.cc:383] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.925163 14010 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.925379 14002 raft_consensus.cc:738] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.925601 14010 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.926239 14003 ts_tablet_manager.cc:1428] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250623 01:58:14.925948 14002 consensus_queue.cc:260] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.927038 14003 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:14.927341 14010 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:14.927906 14002 ts_tablet_manager.cc:1428] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250623 01:58:14.928758 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:14.929498 13811 raft_consensus.cc:2391] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:14.930281 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.928735 14002 tablet_bootstrap.cc:492] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:14.932906 14003 tablet_bootstrap.cc:654] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.924481 14009 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.936132 13925 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:14.938406 14010 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.952116 14081 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.952600 14081 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.955614 14002 tablet_bootstrap.cc:654] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:14.956880 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:14.957635 13885 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.961793 14104 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.962512 14104 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.964841 14104 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.966634 14010 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:14.966894 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:14.968058 14081 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.969494 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:14.970024 14003 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:14.970422 14003 ts_tablet_manager.cc:1397] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.016s
I20250623 01:58:14.970482 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:14.970731 13779 leader_election.cc:343] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.971165 13954 raft_consensus.cc:2391] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:14.972297 13854 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:14.972898 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.975641 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:14.976202 13954 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.973565 14104 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.974699 14117 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:14.977545 13885 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:14.976153 13887 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:14.972833 14003 raft_consensus.cc:357] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.977581 14117 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.978615 14003 raft_consensus.cc:383] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:14.979660 14003 raft_consensus.cc:738] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:14.980203 14009 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:14.980638 14003 consensus_queue.cc:260] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.982909 14003 ts_tablet_manager.cc:1428] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250623 01:58:14.982939 13776 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:14.983775 13776 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:14.984287 14081 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:14.984885 14033 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:14.985211 14033 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:14.985551 14033 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:14.985906 13954 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:14.986629 14003 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:14.987116 14117 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:14.988013 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:14.989267 14081 raft_consensus.cc:695] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
W20250623 01:58:14.989427 13925 leader_election.cc:343] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:14.990108 14081 consensus_queue.cc:237] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.992601 14033 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:14.995564 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:14.997367 14004 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:14.997794 14004 ts_tablet_manager.cc:1397] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.105s user 0.016s sys 0.015s
I20250623 01:58:15.000227 14004 raft_consensus.cc:357] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.000886 14004 raft_consensus.cc:383] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 01:58:15.001534 13926 leader_election.cc:343] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.000252 14104 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.001979 13926 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.002259 14033 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.002892 14117 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.001143 14004 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.004581 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:15.005018 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:15.005230 13886 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.005805 13954 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.005085 14004 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.008352 14004 ts_tablet_manager.cc:1428] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20250623 01:58:15.009205 14004 tablet_bootstrap.cc:492] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.011603 14033 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.005153 14082 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250623 01:58:15.018723 14004 tablet_bootstrap.cc:654] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.028247 13954 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:15.033600 14002 tablet_bootstrap.cc:492] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.034047 14002 ts_tablet_manager.cc:1397] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.007s
I20250623 01:58:15.034638 13779 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:15.035450 13886 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:15.035524 14009 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:15.036104 14033 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:15.036576 14033 raft_consensus.cc:695] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:15.040575 14002 raft_consensus.cc:357] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.037810 14033 consensus_queue.cc:237] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.041222 14002 raft_consensus.cc:383] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.043875 14002 raft_consensus.cc:738] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.044378 14002 consensus_queue.cc:260] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.047577 14002 ts_tablet_manager.cc:1428] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250623 01:58:15.048966 14002 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.051841 13708 catalog_manager.cc:5582] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:15.088337 14002 tablet_bootstrap.cc:654] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.101747 14003 tablet_bootstrap.cc:654] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.111325 13709 catalog_manager.cc:5582] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:15.114599 14004 tablet_bootstrap.cc:492] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.115078 14004 ts_tablet_manager.cc:1397] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.005s
I20250623 01:58:15.117746 14114 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.117995 14004 raft_consensus.cc:357] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.118168 14114 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.118712 14004 raft_consensus.cc:383] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.120733 14004 raft_consensus.cc:738] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.121650 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.121493 14004 consensus_queue.cc:260] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.122359 13811 raft_consensus.cc:2391] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:15.123663 14004 ts_tablet_manager.cc:1428] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250623 01:58:15.124532 14004 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.125007 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.126663 14003 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.127068 14003 ts_tablet_manager.cc:1397] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.002s
I20250623 01:58:15.128091 13854 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:15.128880 14104 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.127000 14114 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.132845 14003 raft_consensus.cc:357] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.133471 14003 raft_consensus.cc:383] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.133702 14003 raft_consensus.cc:738] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.134302 14003 consensus_queue.cc:260] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.138607 14004 tablet_bootstrap.cc:654] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.140719 14104 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.143225 14003 ts_tablet_manager.cc:1428] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250623 01:58:15.144389 14003 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.147307 14004 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.147715 14004 ts_tablet_manager.cc:1397] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.006s
I20250623 01:58:15.150230 14004 raft_consensus.cc:357] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.150826 14004 raft_consensus.cc:383] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.151059 14004 raft_consensus.cc:738] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.151633 14004 consensus_queue.cc:260] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.152241 14002 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.152665 14002 ts_tablet_manager.cc:1397] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.007s
I20250623 01:58:15.153614 14004 ts_tablet_manager.cc:1428] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250623 01:58:15.154413 14004 tablet_bootstrap.cc:492] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.154322 14104 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.155053 14002 raft_consensus.cc:357] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.155628 14002 raft_consensus.cc:383] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.156837 14002 raft_consensus.cc:738] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.157428 14002 consensus_queue.cc:260] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.159575 14002 ts_tablet_manager.cc:1428] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250623 01:58:15.161109 14003 tablet_bootstrap.cc:654] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.162218 14002 tablet_bootstrap.cc:492] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.168496 14004 tablet_bootstrap.cc:654] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.168824 14002 tablet_bootstrap.cc:654] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.170246 14152 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.170753 14152 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.175314 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.175729 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.175910 13886 raft_consensus.cc:2466] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:15.176278 13956 raft_consensus.cc:2466] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:15.177011 14152 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.176951 13776 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:15.177891 14152 raft_consensus.cc:2802] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.180641 14152 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.181856 14152 raft_consensus.cc:3058] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.184427 14003 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.184876 14003 ts_tablet_manager.cc:1397] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.001s
I20250623 01:58:15.188014 14003 raft_consensus.cc:357] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.195564 14003 raft_consensus.cc:383] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.195919 14003 raft_consensus.cc:738] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.196630 14003 consensus_queue.cc:260] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.198388 14010 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.194998 14104 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250623 01:58:15.200459 14152 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.214289 14003 ts_tablet_manager.cc:1428] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20250623 01:58:15.215157 14003 tablet_bootstrap.cc:492] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.216490 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:15.217039 13886 raft_consensus.cc:3058] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.218416 14152 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.226086 14010 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:15.240942 14033 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.241992 14002 tablet_bootstrap.cc:492] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.242348 14002 ts_tablet_manager.cc:1397] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.080s user 0.021s sys 0.006s
I20250623 01:58:15.231611 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:15.243482 13961 raft_consensus.cc:3058] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.241461 14033 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.248042 14033 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.247975 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.250402 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.251243 13956 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:15.253026 14003 tablet_bootstrap.cc:654] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.253459 13779 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:15.255065 13961 raft_consensus.cc:2466] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
W20250623 01:58:15.255594 13776 leader_election.cc:343] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.247996 14004 tablet_bootstrap.cc:492] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.256496 14033 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.256459 13779 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:15.256923 14033 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.257239 14033 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.257493 14152 raft_consensus.cc:2802] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:15.258248 14152 raft_consensus.cc:695] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:15.259091 14152 consensus_queue.cc:237] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.259424 14002 raft_consensus.cc:357] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.260011 14002 raft_consensus.cc:383] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.260267 14002 raft_consensus.cc:738] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.261006 14004 ts_tablet_manager.cc:1397] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.107s user 0.016s sys 0.014s
I20250623 01:58:15.256114 13886 raft_consensus.cc:2466] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:15.275724 14033 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.260871 14002 consensus_queue.cc:260] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.277791 14033 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.277890 14002 ts_tablet_manager.cc:1428] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20250623 01:58:15.278283 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:15.278210 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:15.278913 13961 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.279170 14002 tablet_bootstrap.cc:492] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
W20250623 01:58:15.280181 13776 leader_election.cc:343] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.280505 14004 raft_consensus.cc:357] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.281121 14004 raft_consensus.cc:383] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.281399 14004 raft_consensus.cc:738] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.282016 14004 consensus_queue.cc:260] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.283942 14004 ts_tablet_manager.cc:1428] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.023s user 0.002s sys 0.002s
I20250623 01:58:15.284739 14004 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.285048 13961 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:15.286090 13779 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:15.287065 14033 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:15.287644 14033 raft_consensus.cc:695] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:15.288376 14033 consensus_queue.cc:237] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:15.303319 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:15.305886 14004 tablet_bootstrap.cc:654] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.305233 14002 tablet_bootstrap.cc:654] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.305745 13708 catalog_manager.cc:5582] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:15.309245 14003 tablet_bootstrap.cc:492] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.309684 14003 ts_tablet_manager.cc:1397] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.095s user 0.018s sys 0.001s
I20250623 01:58:15.311978 14003 raft_consensus.cc:357] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.312994 14003 raft_consensus.cc:383] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.313237 14003 raft_consensus.cc:738] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.314087 14003 consensus_queue.cc:260] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.325913 13709 catalog_manager.cc:5582] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:15.329721 14003 ts_tablet_manager.cc:1428] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250623 01:58:15.335572 14004 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.335922 14004 ts_tablet_manager.cc:1397] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.002s
I20250623 01:58:15.338878 14004 raft_consensus.cc:357] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.339489 14004 raft_consensus.cc:383] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.339897 14004 raft_consensus.cc:738] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.340276 14004 consensus_queue.cc:260] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.340951 14003 tablet_bootstrap.cc:492] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.342041 14004 ts_tablet_manager.cc:1428] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250623 01:58:15.343071 14004 tablet_bootstrap.cc:492] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.348835 14004 tablet_bootstrap.cc:654] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.366711 14002 tablet_bootstrap.cc:492] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.367148 14002 ts_tablet_manager.cc:1397] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.088s user 0.025s sys 0.018s
I20250623 01:58:15.369148 14004 tablet_bootstrap.cc:492] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.369544 14004 ts_tablet_manager.cc:1397] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.014s
I20250623 01:58:15.370029 14002 raft_consensus.cc:357] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.370602 14003 tablet_bootstrap.cc:654] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.370723 14002 raft_consensus.cc:383] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.371002 14002 raft_consensus.cc:738] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.371495 14004 raft_consensus.cc:357] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.371747 14002 consensus_queue.cc:260] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.372530 14004 raft_consensus.cc:383] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.372902 14004 raft_consensus.cc:738] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.372932 14114 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.373453 14114 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.373678 14004 consensus_queue.cc:260] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.375195 14002 ts_tablet_manager.cc:1428] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250623 01:58:15.376109 14002 tablet_bootstrap.cc:492] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.378126 14114 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.379348 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.379976 13961 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:15.382136 14114 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.385565 14114 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.387624 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.387893 14003 tablet_bootstrap.cc:492] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.388280 14003 ts_tablet_manager.cc:1397] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.015s
I20250623 01:58:15.383065 13854 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:15.388813 13852 leader_election.cc:343] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.388999 14104 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.389374 14104 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.389714 14104 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.390640 14003 raft_consensus.cc:357] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.391113 14003 raft_consensus.cc:383] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.391355 14003 raft_consensus.cc:738] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.392058 14003 consensus_queue.cc:260] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.392800 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.393455 13961 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:15.394048 14114 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.382447 14002 tablet_bootstrap.cc:654] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.388139 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.394464 13854 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:15.395720 13852 leader_election.cc:343] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.384763 14004 ts_tablet_manager.cc:1428] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250623 01:58:15.396854 14114 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.397068 14004 tablet_bootstrap.cc:492] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.397213 14114 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.397418 14003 ts_tablet_manager.cc:1428] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20250623 01:58:15.398135 14114 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.398306 14003 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.403987 14004 tablet_bootstrap.cc:654] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.404910 14114 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.406059 14104 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.410491 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:15.411733 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:15.412287 13961 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:58:15.412354 13852 leader_election.cc:343] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.412459 14003 tablet_bootstrap.cc:654] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.413180 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:15.414795 13852 leader_election.cc:343] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.414530 14114 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.415149 14104 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.419291 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:15.419557 14002 tablet_bootstrap.cc:492] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.419723 13954 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.419996 14002 ts_tablet_manager.cc:1397] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.001s
I20250623 01:58:15.422547 14004 tablet_bootstrap.cc:492] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.422425 13961 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:15.422435 14002 raft_consensus.cc:357] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.423295 14002 raft_consensus.cc:383] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.423586 14002 raft_consensus.cc:738] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.423666 14004 ts_tablet_manager.cc:1397] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.001s
I20250623 01:58:15.423801 13854 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:15.424182 14002 consensus_queue.cc:260] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.425025 14104 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:15.425485 14104 raft_consensus.cc:695] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:15.426388 13954 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:15.426898 14002 ts_tablet_manager.cc:1428] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20250623 01:58:15.426461 14104 consensus_queue.cc:237] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.427389 13854 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:15.428406 14002 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.428359 14057 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:15.430541 14004 raft_consensus.cc:357] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.431228 14004 raft_consensus.cc:383] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.431520 14004 raft_consensus.cc:738] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.432322 14004 consensus_queue.cc:260] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.433060 14003 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.433566 14003 ts_tablet_manager.cc:1397] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.002s
I20250623 01:58:15.434872 14004 ts_tablet_manager.cc:1428] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250623 01:58:15.435993 14004 tablet_bootstrap.cc:492] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.436445 14003 raft_consensus.cc:357] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.437053 14003 raft_consensus.cc:383] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.437331 14003 raft_consensus.cc:738] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.437960 14003 consensus_queue.cc:260] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.441372 14003 ts_tablet_manager.cc:1428] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250623 01:58:15.442518 14003 tablet_bootstrap.cc:492] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.442560 14002 tablet_bootstrap.cc:654] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.448628 14004 tablet_bootstrap.cc:654] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.447896 14010 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.451785 14003 tablet_bootstrap.cc:654] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.451123 14010 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.456156 14010 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:15.459168 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:15.461282 13925 leader_election.cc:343] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.461506 14152 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.462173 14152 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.463891 14057 raft_consensus.cc:695] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:15.464753 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.477226 14033 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.464860 14057 consensus_queue.cc:237] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.477675 14033 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.465080 14152 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.465960 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.479429 14033 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.480151 14002 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.480654 14002 ts_tablet_manager.cc:1397] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.006s
I20250623 01:58:15.481444 14003 tablet_bootstrap.cc:492] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.481856 14003 ts_tablet_manager.cc:1397] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.003s
I20250623 01:58:15.482522 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.483132 13961 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:15.483084 14002 raft_consensus.cc:357] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.483737 14002 raft_consensus.cc:383] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.482595 13809 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:15.483387 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.484180 13779 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:15.469276 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:15.485616 13776 leader_election.cc:343] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.484165 14002 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.486739 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.487319 14003 raft_consensus.cc:357] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.487736 14152 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.487949 14003 raft_consensus.cc:383] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.488222 14003 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.488762 14003 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.490675 14003 ts_tablet_manager.cc:1428] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250623 01:58:15.490876 14033 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.491716 14003 tablet_bootstrap.cc:492] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.492902 14033 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.493425 14033 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.488327 14002 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.495872 14002 ts_tablet_manager.cc:1428] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20250623 01:58:15.496929 14002 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.503206 14174 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.503664 14174 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.504187 14002 tablet_bootstrap.cc:654] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.505451 14174 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.506726 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.507484 13813 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:15.508340 14082 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.508555 13926 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:15.508672 14033 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.509606 14010 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.509941 14010 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.510069 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.510272 14010 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.510641 13954 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:15.511412 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:15.511757 13854 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:15.511945 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:15.513044 14033 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.513717 14174 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250623 01:58:15.513998 13776 leader_election.cc:343] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.514053 14174 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.516902 14174 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.517072 14010 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.518530 13961 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:15.521673 13779 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.523079 14174 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.523581 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:15.523609 14152 consensus_queue.cc:1035] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:15.524761 14081 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250623 01:58:15.525939 13925 leader_election.cc:343] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.526510 14010 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:15.529301 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:15.530372 13813 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:15.531286 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:15.532094 13809 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.533718 14002 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.533016 13926 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.534214 14002 ts_tablet_manager.cc:1397] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.006s
I20250623 01:58:15.535240 14010 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.536000 14082 consensus_queue.cc:1035] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.536814 14002 raft_consensus.cc:357] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.537544 14002 raft_consensus.cc:383] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.537810 14002 raft_consensus.cc:738] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.538455 14002 consensus_queue.cc:260] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.539356 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:15.539876 13961 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.545455 14002 ts_tablet_manager.cc:1428] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250623 01:58:15.546305 14002 tablet_bootstrap.cc:492] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.548256 14004 tablet_bootstrap.cc:492] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.553442 14004 ts_tablet_manager.cc:1397] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.118s user 0.012s sys 0.002s
I20250623 01:58:15.556409 14003 tablet_bootstrap.cc:654] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.549886 13961 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:15.560093 13854 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:15.560606 14002 tablet_bootstrap.cc:654] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.560912 14104 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:15.561368 14104 raft_consensus.cc:695] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:15.562181 14104 consensus_queue.cc:237] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.550251 14174 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.575095 14152 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.576334 14082 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.586874 14152 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.590078 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.590574 14152 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:15.591903 13776 leader_election.cc:343] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.594672 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.595494 13809 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:15.596202 13779 leader_election.cc:343] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.596724 13779 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.597600 14082 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.626914 14003 tablet_bootstrap.cc:492] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.627285 14003 ts_tablet_manager.cc:1397] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.136s user 0.009s sys 0.000s
I20250623 01:58:15.629464 14003 raft_consensus.cc:357] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.630105 14003 raft_consensus.cc:383] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.630363 14003 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.633141 14002 tablet_bootstrap.cc:492] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.634727 14003 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.636868 14002 ts_tablet_manager.cc:1397] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.091s user 0.026s sys 0.014s
I20250623 01:58:15.571308 14004 raft_consensus.cc:357] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.638435 14004 raft_consensus.cc:383] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.638777 14004 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.639158 14002 raft_consensus.cc:357] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.639513 14004 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.640304 14002 raft_consensus.cc:383] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.640623 14002 raft_consensus.cc:738] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.641516 14002 consensus_queue.cc:260] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.643605 14002 ts_tablet_manager.cc:1428] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20250623 01:58:15.644408 14002 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.639531 14003 ts_tablet_manager.cc:1428] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250623 01:58:15.646670 14003 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.650184 14004 ts_tablet_manager.cc:1428] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.095s user 0.004s sys 0.003s
I20250623 01:58:15.650486 14002 tablet_bootstrap.cc:654] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.651028 14004 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.651470 14003 tablet_bootstrap.cc:654] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.657130 14057 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.657601 14057 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.660161 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.661022 14057 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.661886 14004 tablet_bootstrap.cc:654] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.661947 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.662722 13961 raft_consensus.cc:2391] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:15.665940 13854 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:15.666877 14057 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.668088 13710 catalog_manager.cc:5582] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:15.670352 13710 catalog_manager.cc:5582] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:15.675271 14057 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.676113 14003 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.676503 14003 ts_tablet_manager.cc:1397] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.003s
I20250623 01:58:15.677515 14004 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.677929 14004 ts_tablet_manager.cc:1397] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.006s
I20250623 01:58:15.679270 14003 raft_consensus.cc:357] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.679888 14003 raft_consensus.cc:383] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.680176 14003 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.679971 14004 raft_consensus.cc:357] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.680586 14004 raft_consensus.cc:383] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.680861 14004 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.680958 14003 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.681536 14004 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.683710 14004 ts_tablet_manager.cc:1428] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20250623 01:58:15.683712 14003 ts_tablet_manager.cc:1428] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250623 01:58:15.684589 14003 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.684589 14004 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.690409 14003 tablet_bootstrap.cc:654] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.695185 14004 tablet_bootstrap.cc:654] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.709235 14003 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.709884 14003 ts_tablet_manager.cc:1397] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.006s
I20250623 01:58:15.715814 14003 raft_consensus.cc:357] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.716950 14117 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.717633 14117 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.720064 14117 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:15.722069 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.723500 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.723830 13809 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.724248 14003 raft_consensus.cc:383] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.724617 14003 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
W20250623 01:58:15.723666 13926 leader_election.cc:343] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
W20250623 01:58:15.725603 13925 leader_election.cc:343] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.725417 14003 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.726037 13925 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.728502 14010 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.735453 14003 ts_tablet_manager.cc:1428] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
I20250623 01:58:15.736320 14003 tablet_bootstrap.cc:492] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.738430 14117 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.739001 14117 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.743175 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.743777 14003 tablet_bootstrap.cc:654] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:15.745059 13926 leader_election.cc:343] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.746080 14117 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:15.749271 13711 catalog_manager.cc:5582] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:15.753688 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:15.758311 13925 leader_election.cc:343] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.758889 13925 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.762224 14004 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.762684 14004 ts_tablet_manager.cc:1397] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.078s user 0.027s sys 0.007s
I20250623 01:58:15.762246 14003 tablet_bootstrap.cc:492] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.764848 14002 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.765224 14003 ts_tablet_manager.cc:1397] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.012s
I20250623 01:58:15.765614 14004 raft_consensus.cc:357] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.765585 14002 ts_tablet_manager.cc:1397] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.121s user 0.037s sys 0.058s
I20250623 01:58:15.766324 14004 raft_consensus.cc:383] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.766626 14004 raft_consensus.cc:738] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.767632 14004 consensus_queue.cc:260] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.767787 14003 raft_consensus.cc:357] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.768424 14003 raft_consensus.cc:383] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.768708 14003 raft_consensus.cc:738] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.768707 14002 raft_consensus.cc:357] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.769539 14002 raft_consensus.cc:383] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.769832 14004 ts_tablet_manager.cc:1428] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250623 01:58:15.769563 14003 consensus_queue.cc:260] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.769948 14002 raft_consensus.cc:738] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.770632 14010 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:15.771003 14004 tablet_bootstrap.cc:492] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.770969 14002 consensus_queue.cc:260] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.772689 14003 ts_tablet_manager.cc:1428] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250623 01:58:15.773563 14003 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.773936 14002 ts_tablet_manager.cc:1428] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250623 01:58:15.775079 14002 tablet_bootstrap.cc:492] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.780233 14004 tablet_bootstrap.cc:654] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.780911 14003 tablet_bootstrap.cc:654] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.781111 14002 tablet_bootstrap.cc:654] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.792438 14004 tablet_bootstrap.cc:492] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:15.793435 14004 ts_tablet_manager.cc:1397] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.005s
I20250623 01:58:15.793728 14003 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.794014 14002 tablet_bootstrap.cc:492] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.794138 14003 ts_tablet_manager.cc:1397] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20250623 01:58:15.794437 14002 ts_tablet_manager.cc:1397] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20250623 01:58:15.796097 14004 raft_consensus.cc:357] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.796337 14003 raft_consensus.cc:357] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.796931 14004 raft_consensus.cc:383] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.796965 14003 raft_consensus.cc:383] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.796772 14002 raft_consensus.cc:357] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.797526 14003 raft_consensus.cc:738] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.797650 14002 raft_consensus.cc:383] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.797524 14004 raft_consensus.cc:738] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.798097 14002 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.798430 14003 consensus_queue.cc:260] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.798600 14004 consensus_queue.cc:260] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.799947 14002 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.801225 14003 ts_tablet_manager.cc:1428] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250623 01:58:15.802122 14003 tablet_bootstrap.cc:492] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:15.802117 14002 ts_tablet_manager.cc:1428] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250623 01:58:15.803153 14002 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.808957 14003 tablet_bootstrap.cc:654] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.809119 14002 tablet_bootstrap.cc:654] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.811630 14082 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.815250 14002 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:15.815681 14002 ts_tablet_manager.cc:1397] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20250623 01:58:15.817996 14002 raft_consensus.cc:357] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.818607 14002 raft_consensus.cc:383] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.818852 14002 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.819396 14002 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.821662 14002 ts_tablet_manager.cc:1428] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250623 01:58:15.822077 14033 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.806093 14004 ts_tablet_manager.cc:1428] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20250623 01:58:15.824890 14004 tablet_bootstrap.cc:492] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:15.833348 14002 tablet_bootstrap.cc:492] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:15.835078 14010 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.835654 14010 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.837694 14010 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:15.838526 14057 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.839643 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.839787 14057 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.840214 13809 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:15.841643 13926 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:15.842054 14057 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.842411 14010 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.842787 14010 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.843086 14010 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.847018 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.848731 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.850333 13809 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
W20250623 01:58:15.850764 13925 leader_election.cc:343] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.852455 13852 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:15.853457 14057 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.853819 14057 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.854125 14057 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.854880 14033 consensus_queue.cc:1035] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.887527 14003 tablet_bootstrap.cc:492] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:15.887940 14003 ts_tablet_manager.cc:1397] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.086s user 0.027s sys 0.016s
I20250623 01:58:15.887455 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.891671 14117 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.892657 14117 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.894475 14117 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:15.895105 14187 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.896621 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.896989 14187 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.895880 13959 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
W20250623 01:58:15.898490 13926 leader_election.cc:343] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.898938 14187 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.900036 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.902053 13956 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:15.903218 14082 consensus_queue.cc:1035] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:15.900990 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.904222 13809 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:15.904556 13854 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:15.900553 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.905387 14187 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.905714 14187 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.906039 14187 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.912453 14081 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:15.912961 14081 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:15.900223 13852 consensus_peers.cc:487] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:15.912861 14187 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.914809 14081 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.917265 14117 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:15.917465 13925 leader_election.cc:343] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:15.916857 14187 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.915913 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:15.919934 14117 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.925623 13961 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.927058 14117 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:15.922318 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:15.928087 13954 raft_consensus.cc:2391] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:15.929404 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:15.929421 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:15.930086 13809 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.930274 13813 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:15.935421 13926 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:15.937206 14117 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:15.937592 14117 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:15.937914 14117 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.923496 13925 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.931087 13961 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:15.952812 14004 tablet_bootstrap.cc:654] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.955021 14003 raft_consensus.cc:357] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.955443 14003 raft_consensus.cc:383] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:15.955653 14003 raft_consensus.cc:738] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:15.957067 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.957551 13813 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.957685 13886 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:15.958667 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:15.960453 14002 tablet_bootstrap.cc:654] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.962430 14104 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250623 01:58:15.961702 14003 consensus_queue.cc:260] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.966363 13776 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:15.960700 13854 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:15.966185 14057 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.974752 14104 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:15.975188 14104 raft_consensus.cc:695] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:15.972162 14117 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.982774 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:15.982569 14104 consensus_queue.cc:237] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:15.983314 13811 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:15.987717 14057 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:15.998137 14003 ts_tablet_manager.cc:1428] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.110s user 0.005s sys 0.006s
I20250623 01:58:15.999153 14003 tablet_bootstrap.cc:492] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:16.005668 14003 tablet_bootstrap.cc:654] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:15.966395 14010 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.014147 14081 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.018239 14117 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:16.018069 14081 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.018769 14117 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:16.019840 14117 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.019974 14081 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.020249 14117 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.023622 14010 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:16.023661 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:16.024519 13885 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.025967 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:16.027494 13925 leader_election.cc:343] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.014361 14057 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.028188 13809 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:16.056205 14104 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.056708 14104 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.058619 14104 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.046037 14057 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.059274 14117 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:16.060412 14057 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.061051 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.061105 13811 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:16.064632 14002 tablet_bootstrap.cc:492] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:16.065025 14002 ts_tablet_manager.cc:1397] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.238s user 0.008s sys 0.031s
I20250623 01:58:16.066666 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:16.067240 13812 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.067145 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:16.050020 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:16.069698 13961 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.070580 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:16.072410 13776 leader_election.cc:343] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.073477 13962 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:16.073680 13709 catalog_manager.cc:5582] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:16.080202 14082 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:16.073894 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:16.091411 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:16.092029 13810 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.126516 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:16.127372 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.127959 13954 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:16.131436 13854 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:16.132849 14104 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:16.133194 14104 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:16.133520 14104 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.140678 13961 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:16.149878 14004 tablet_bootstrap.cc:492] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:16.150514 14004 ts_tablet_manager.cc:1397] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.326s user 0.027s sys 0.013s
I20250623 01:58:16.152483 14002 raft_consensus.cc:357] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.153155 14002 raft_consensus.cc:383] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.153456 14002 raft_consensus.cc:738] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.154064 14002 consensus_queue.cc:260] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.125087 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.145970 13885 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:16.156924 13809 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:16.141099 14104 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.146168 13852 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:16.168563 14004 raft_consensus.cc:357] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.171100 14003 tablet_bootstrap.cc:492] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:16.171499 14003 ts_tablet_manager.cc:1397] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.173s user 0.009s sys 0.010s
I20250623 01:58:16.174927 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:16.177608 14003 raft_consensus.cc:357] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.178231 14003 raft_consensus.cc:383] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.178506 14003 raft_consensus.cc:738] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.179188 14003 consensus_queue.cc:260] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.180998 14002 ts_tablet_manager.cc:1428] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.116s user 0.007s sys 0.000s
I20250623 01:58:16.182013 14002 tablet_bootstrap.cc:492] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:16.181052 14003 ts_tablet_manager.cc:1428] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250623 01:58:16.183488 14003 tablet_bootstrap.cc:492] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:16.169471 14004 raft_consensus.cc:383] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.184464 14004 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.185185 14004 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.158214 13959 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
W20250623 01:58:16.191454 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 101830us
I20250623 01:58:16.193552 14187 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:16.205586 14004 ts_tablet_manager.cc:1428] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
W20250623 01:58:16.207840 13926 leader_election.cc:343] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.210809 13779 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:16.211551 14081 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:16.211973 14004 tablet_bootstrap.cc:492] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:16.211928 14081 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:16.212385 14081 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.213819 14187 raft_consensus.cc:695] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:16.214649 14187 consensus_queue.cc:237] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.206174 14174 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20250623 01:58:16.203476 13852 leader_election.cc:343] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.237787 14104 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.239714 13812 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:16.246337 14174 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 01:58:16.253808 13852 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:16.254559 14174 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.256913 13709 catalog_manager.cc:5582] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:16.256929 13925 leader_election.cc:343] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.259068 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:16.259635 13956 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.266770 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:16.263784 13926 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:16.259081 13925 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:16.272392 14216 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:16.274273 14212 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:16.274775 14212 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:16.275530 14212 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.267747 13810 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:16.279572 14117 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:16.290951 14174 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:16.282282 14081 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.302800 14081 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.298067 13926 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:16.303333 14216 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:16.304564 14117 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:16.305068 14117 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:16.305861 14117 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.309103 14002 tablet_bootstrap.cc:654] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.309381 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:16.309859 13961 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.311491 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:16.297567 13852 leader_election.cc:343] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.315968 13961 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:16.317782 13779 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:16.320425 14004 tablet_bootstrap.cc:654] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:16.329684 13776 leader_election.cc:343] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.337962 13956 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:16.347868 14003 tablet_bootstrap.cc:654] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.332687 14081 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:16.349391 13854 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:58:16.357537 14219 log.cc:927] Time spent T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20250623 01:58:16.360955 14081 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:16.365796 14081 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.366690 13709 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:16.368393 14057 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:16.369133 14057 raft_consensus.cc:695] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:16.370087 14057 consensus_queue.cc:237] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.368461 13709 catalog_manager.cc:5582] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:16.376401 14002 tablet_bootstrap.cc:492] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:16.435765 14002 ts_tablet_manager.cc:1397] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.254s user 0.013s sys 0.026s
I20250623 01:58:16.439292 14003 tablet_bootstrap.cc:492] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:16.439744 14003 ts_tablet_manager.cc:1397] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.256s user 0.008s sys 0.003s
I20250623 01:58:16.442102 14003 raft_consensus.cc:357] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.442760 14003 raft_consensus.cc:383] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.443562 14003 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.444098 14003 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:16.399181 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:16.446367 14003 ts_tablet_manager.cc:1428] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20250623 01:58:16.447476 14002 raft_consensus.cc:357] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.447965 14002 raft_consensus.cc:383] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.448197 14002 raft_consensus.cc:738] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.448661 14002 consensus_queue.cc:260] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.435194 14004 tablet_bootstrap.cc:492] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:16.449820 14004 ts_tablet_manager.cc:1397] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.241s user 0.020s sys 0.022s
I20250623 01:58:16.398329 14104 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20250623 01:58:16.453061 14002 ts_tablet_manager.cc:1428] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.017s user 0.001s sys 0.005s
I20250623 01:58:16.453948 14002 tablet_bootstrap.cc:492] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:16.454049 14004 raft_consensus.cc:357] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.455134 14004 raft_consensus.cc:383] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.456137 14004 raft_consensus.cc:738] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.456925 14004 consensus_queue.cc:260] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.459486 14004 ts_tablet_manager.cc:1428] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20250623 01:58:16.459672 14002 tablet_bootstrap.cc:654] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.460368 14004 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:16.478201 14003 tablet_bootstrap.cc:492] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:16.485456 14003 tablet_bootstrap.cc:654] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.488502 14216 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:16.492043 14002 tablet_bootstrap.cc:492] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:16.492487 14002 ts_tablet_manager.cc:1397] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.009s
I20250623 01:58:16.489244 14004 tablet_bootstrap.cc:654] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.494971 14002 raft_consensus.cc:357] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.497412 14002 raft_consensus.cc:383] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.497768 14002 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.498394 14002 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.500870 14002 ts_tablet_manager.cc:1428] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250623 01:58:16.501757 14002 tablet_bootstrap.cc:492] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:16.507941 14003 tablet_bootstrap.cc:492] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:16.508510 14003 ts_tablet_manager.cc:1397] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.004s
I20250623 01:58:16.513945 14004 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:16.514322 14004 ts_tablet_manager.cc:1397] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.054s user 0.018s sys 0.005s
I20250623 01:58:16.514735 14217 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.515187 14217 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.516680 14004 raft_consensus.cc:357] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.517325 14004 raft_consensus.cc:383] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.518342 14004 raft_consensus.cc:738] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.519111 14004 consensus_queue.cc:260] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.519759 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:16.520473 14216 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.521354 14216 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.521816 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.522336 13961 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:16.525542 14216 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.525605 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:16.519001 14217 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.527153 13886 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:16.533102 14002 tablet_bootstrap.cc:654] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.533849 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.534654 13961 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:16.535506 14217 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:16.535907 14217 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:16.535658 14004 ts_tablet_manager.cc:1428] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20250623 01:58:16.537410 14004 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:16.552019 13710 catalog_manager.cc:5582] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:16.539397 14003 raft_consensus.cc:357] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.536273 14217 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.554284 14004 tablet_bootstrap.cc:654] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.554585 14003 raft_consensus.cc:383] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.555097 14003 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.555773 14003 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.527288 14104 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.536152 13854 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:16.546111 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:16.545998 13709 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:16.566491 14003 ts_tablet_manager.cc:1428] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.058s user 0.005s sys 0.003s
I20250623 01:58:16.567373 14003 tablet_bootstrap.cc:492] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:16.564401 14104 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.569088 14057 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:16.569501 14057 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:16.569624 14104 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.569861 14057 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.572345 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.575763 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:16.576483 13811 raft_consensus.cc:2391] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:16.577492 13852 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:16.578405 14104 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.580232 14004 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:16.580700 14004 ts_tablet_manager.cc:1397] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.009s
I20250623 01:58:16.584901 14217 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.587617 13810 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:16.593474 14057 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.596285 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:16.597529 14217 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.597445 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:16.598001 13886 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:16.598073 13954 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.606580 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:16.607311 13811 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:16.608418 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:16.609764 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:16.610510 14081 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 01:58:16.595892 14057 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.617250 14004 raft_consensus.cc:357] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.617995 14004 raft_consensus.cc:383] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.618317 14004 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.618520 14104 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:16.619129 14004 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.621392 13954 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:16.619853 14003 tablet_bootstrap.cc:654] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.622368 13854 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:16.621042 14002 tablet_bootstrap.cc:492] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:16.623353 14002 ts_tablet_manager.cc:1397] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.122s user 0.028s sys 0.032s
I20250623 01:58:16.623354 14057 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:16.624413 14057 raft_consensus.cc:695] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:16.625197 14057 consensus_queue.cc:237] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.626991 14002 raft_consensus.cc:357] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.627889 14002 raft_consensus.cc:383] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.628170 14002 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.628912 14002 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.625828 14004 ts_tablet_manager.cc:1428] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.045s user 0.006s sys 0.003s
I20250623 01:58:16.631217 14004 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:16.636404 14002 ts_tablet_manager.cc:1428] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20250623 01:58:16.638394 14002 tablet_bootstrap.cc:492] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:16.648340 14003 tablet_bootstrap.cc:492] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:16.648870 14003 ts_tablet_manager.cc:1397] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.011s
I20250623 01:58:16.650852 14003 raft_consensus.cc:357] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.651304 14003 raft_consensus.cc:383] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.651504 14003 raft_consensus.cc:738] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.652001 14003 consensus_queue.cc:260] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.653668 14002 tablet_bootstrap.cc:654] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.654107 14004 tablet_bootstrap.cc:654] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.654052 14003 ts_tablet_manager.cc:1428] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20250623 01:58:16.655129 14003 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:16.662315 14003 tablet_bootstrap.cc:654] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.667905 14002 tablet_bootstrap.cc:492] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:16.668474 14002 ts_tablet_manager.cc:1397] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.009s
I20250623 01:58:16.674160 13709 catalog_manager.cc:5582] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:16.676090 14004 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:16.677340 14004 ts_tablet_manager.cc:1397] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.008s
I20250623 01:58:16.677028 14002 raft_consensus.cc:357] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.677830 14002 raft_consensus.cc:383] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.678153 14002 raft_consensus.cc:738] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.678253 14003 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:16.678740 14003 ts_tablet_manager.cc:1397] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.001s
I20250623 01:58:16.678853 14002 consensus_queue.cc:260] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.679586 14004 raft_consensus.cc:357] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.680152 14004 raft_consensus.cc:383] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.680403 14004 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.680884 14003 raft_consensus.cc:357] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.681041 14004 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.681587 14003 raft_consensus.cc:383] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.681877 14003 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.686563 14003 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.687180 14002 ts_tablet_manager.cc:1428] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.018s user 0.004s sys 0.006s
I20250623 01:58:16.687477 14004 ts_tablet_manager.cc:1428] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20250623 01:58:16.688095 14002 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:16.688467 14004 tablet_bootstrap.cc:492] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:16.688548 14003 ts_tablet_manager.cc:1428] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20250623 01:58:16.689460 14003 tablet_bootstrap.cc:492] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:16.690277 14033 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.690748 14033 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.693501 14033 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.696013 14004 tablet_bootstrap.cc:654] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.700641 14003 tablet_bootstrap.cc:654] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.701699 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:16.706519 14002 tablet_bootstrap.cc:654] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:16.710628 13776 leader_election.cc:343] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.712313 14212 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:16.717140 14004 tablet_bootstrap.cc:492] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:16.717551 14004 ts_tablet_manager.cc:1397] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.004s
I20250623 01:58:16.717479 14057 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:16.719386 14003 tablet_bootstrap.cc:492] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:16.719733 14003 ts_tablet_manager.cc:1397] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.008s
I20250623 01:58:16.719125 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.721968 14003 raft_consensus.cc:357] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.722647 14003 raft_consensus.cc:383] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.722630 13954 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:16.723078 14003 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.726297 14004 raft_consensus.cc:357] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.726996 14004 raft_consensus.cc:383] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.727283 14004 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.723692 14003 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.727888 14004 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.729178 14002 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:16.729430 14003 ts_tablet_manager.cc:1428] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250623 01:58:16.729650 14002 ts_tablet_manager.cc:1397] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.000s
I20250623 01:58:16.729081 13779 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:16.732158 14033 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:16.732522 14033 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:16.732336 14002 raft_consensus.cc:357] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.732877 14033 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.733000 14002 raft_consensus.cc:383] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.733280 14002 raft_consensus.cc:738] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.733882 14002 consensus_queue.cc:260] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.734763 14004 ts_tablet_manager.cc:1428] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20250623 01:58:16.735646 14004 tablet_bootstrap.cc:492] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:16.735862 14002 ts_tablet_manager.cc:1428] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250623 01:58:16.736783 14002 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:16.746812 14003 tablet_bootstrap.cc:492] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:16.747915 14057 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:16.751885 14002 tablet_bootstrap.cc:654] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.753835 14212 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.754257 14212 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.741048 14004 tablet_bootstrap.cc:654] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.757287 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:16.758275 14212 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:16.761835 13925 leader_election.cc:343] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.766552 14003 tablet_bootstrap.cc:654] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.771343 14212 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:16.773964 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:16.775470 13926 leader_election.cc:343] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.776067 14004 tablet_bootstrap.cc:492] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:16.776022 13926 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:16.776896 14004 ts_tablet_manager.cc:1397] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.003s
I20250623 01:58:16.777031 14212 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:16.780416 14004 raft_consensus.cc:357] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.782207 14004 raft_consensus.cc:383] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.782646 14004 raft_consensus.cc:738] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.783921 14004 consensus_queue.cc:260] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.785950 14217 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.786523 14217 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.787132 14003 tablet_bootstrap.cc:492] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:16.788605 14217 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.786499 14004 ts_tablet_manager.cc:1428] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20250623 01:58:16.789554 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:16.790205 14004 tablet_bootstrap.cc:492] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:16.790395 14212 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:16.794782 14003 ts_tablet_manager.cc:1397] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.005s
I20250623 01:58:16.797560 14004 tablet_bootstrap.cc:654] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:16.797485 14003 raft_consensus.cc:357] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.798206 14003 raft_consensus.cc:383] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.798074 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.798584 14003 raft_consensus.cc:738] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.809564 14003 consensus_queue.cc:260] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.813652 14104 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.814293 14104 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.816618 14104 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.818022 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:16.820410 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:16.821132 13852 leader_election.cc:343] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.821154 13961 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:16.823244 13854 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:16.824091 14003 ts_tablet_manager.cc:1428] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250623 01:58:16.825145 14187 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.829794 14187 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.825722 14216 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:16.831894 14216 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:16.832378 14216 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.826157 14057 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:16.834812 14057 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.835284 14033 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.837364 14057 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.840066 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.840555 14187 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.840828 13961 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:16.842676 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:16.843533 13807 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:16.845832 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:16.847843 13852 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:16.847918 13779 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:16.848945 14217 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:16.853533 14187 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:16.854063 14187 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:16.854488 14187 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.857956 14104 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:16.859632 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:16.860541 13961 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:16.864534 14187 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.867448 14003 tablet_bootstrap.cc:492] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:16.866101 13854 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:16.879674 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:16.880326 13885 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.876953 14104 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.891810 14117 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:16.872329 14216 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.896472 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:16.897960 13812 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:58:16.900372 13852 leader_election.cc:343] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:16.906213 14033 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.909389 13885 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:16.910621 13776 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:16.911965 14033 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:16.913997 14033 raft_consensus.cc:695] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:16.914867 14033 consensus_queue.cc:237] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.917502 14216 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.919533 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:16.920109 13956 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.932348 13807 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.943262 14004 tablet_bootstrap.cc:492] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:16.943744 14004 ts_tablet_manager.cc:1397] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.154s user 0.041s sys 0.049s
I20250623 01:58:16.945034 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:16.945515 13961 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.956380 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:16.957011 13808 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:16.965399 13956 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:16.967888 14002 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:16.968344 14002 ts_tablet_manager.cc:1397] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.232s user 0.036s sys 0.061s
I20250623 01:58:16.978598 14215 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:16.980247 13854 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:16.981093 14215 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:16.981563 14215 raft_consensus.cc:695] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:16.982322 14215 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.985303 14002 raft_consensus.cc:357] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:16.986668 14002 raft_consensus.cc:383] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:16.986991 14002 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:16.983831 14187 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:16.987715 14002 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.004297 14002 ts_tablet_manager.cc:1428] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.036s user 0.007s sys 0.001s
I20250623 01:58:17.005610 13961 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:17.005602 14002 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:17.012454 14217 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250623 01:58:17.024528 14003 tablet_bootstrap.cc:654] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.026563 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:17.029970 13961 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.043103 13709 catalog_manager.cc:5582] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:17.045226 14254 log.cc:927] Time spent T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250623 01:58:17.043805 13961 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:17.046768 14104 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.050066 13854 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:17.048695 13808 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:17.053382 14187 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:17.053922 14187 raft_consensus.cc:695] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:17.054939 14187 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.059057 14004 raft_consensus.cc:357] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.059715 14004 raft_consensus.cc:383] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.059965 14004 raft_consensus.cc:738] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.061903 14004 consensus_queue.cc:260] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.074926 14033 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.078251 14256 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.083613 14004 ts_tablet_manager.cc:1428] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.140s user 0.006s sys 0.001s
I20250623 01:58:17.084280 14004 tablet_bootstrap.cc:492] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:17.077852 14217 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.088352 14002 tablet_bootstrap.cc:654] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.096148 14256 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.094846 14217 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.114807 14217 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:17.115659 14257 log.cc:927] Time spent T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250623 01:58:17.101466 14256 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:17.123068 14265 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.129058 13711 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:17.131063 13711 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:17.137637 14217 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.123795 14265 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.138180 14217 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.149590 14217 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.154577 14004 tablet_bootstrap.cc:654] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.162412 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:17.163998 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.164655 13886 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:17.166643 14265 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:17.167711 14002 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:17.168112 14002 ts_tablet_manager.cc:1397] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.163s user 0.025s sys 0.000s
I20250623 01:58:17.166195 13925 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:17.170603 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.171165 13888 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
W20250623 01:58:17.171805 14263 log.cc:927] Time spent T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20250623 01:58:17.174760 13776 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:17.175402 14217 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:17.175861 14217 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:17.176381 14217 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.177654 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:17.178296 13959 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:17.178589 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.178728 14002 raft_consensus.cc:357] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.179303 14002 raft_consensus.cc:383] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.179576 14002 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.180186 14002 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.181797 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:17.182672 13925 leader_election.cc:343] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.182916 14002 ts_tablet_manager.cc:1428] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250623 01:58:17.183490 14003 tablet_bootstrap.cc:492] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:17.183822 14002 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:17.183889 14003 ts_tablet_manager.cc:1397] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.317s user 0.010s sys 0.023s
I20250623 01:58:17.179778 14265 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:17.185400 14259 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:17.187075 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:17.189869 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.189926 13779 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:17.190728 14033 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:17.191033 14033 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:17.191246 14033 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:17.191610 14081 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.188370 14265 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:17.192255 14212 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.192029 14081 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.192458 14265 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.192713 14212 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.194033 14081 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.194532 14212 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:17.197729 13926 leader_election.cc:343] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.197667 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.205541 14003 raft_consensus.cc:357] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.206877 14003 raft_consensus.cc:383] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.207187 14003 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.207679 14003 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.202219 14033 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.187845 14217 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.207010 13887 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
W20250623 01:58:17.243693 13925 leader_election.cc:343] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.212226 13926 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:17.285899 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 64450.1us
I20250623 01:58:17.317953 14217 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.323678 13807 raft_consensus.cc:2391] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:17.325105 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:17.328401 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 130686us
I20250623 01:58:17.329397 14033 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.331151 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:17.331686 14002 tablet_bootstrap.cc:654] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.331751 13961 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:17.330798 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:17.332355 14266 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.339161 14003 ts_tablet_manager.cc:1428] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.155s user 0.009s sys 0.022s
I20250623 01:58:17.340986 13961 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:17.342029 14003 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:17.347075 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:17.347887 13776 leader_election.cc:343] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.349437 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:17.349953 13887 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:17.350883 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:17.351387 13888 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.339105 14259 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250623 01:58:17.361582 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:17.362654 13779 leader_election.cc:343] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.366904 13779 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:17.368062 13779 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:17.368203 14271 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.368837 14271 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:17.369231 14265 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.370668 13954 raft_consensus.cc:2391] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:17.370918 14271 raft_consensus.cc:695] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:17.371685 14271 consensus_queue.cc:237] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.372234 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:17.372972 14265 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:17.378954 13926 leader_election.cc:343] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.379473 13926 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:17.381954 14004 tablet_bootstrap.cc:492] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:17.382550 14004 ts_tablet_manager.cc:1397] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.298s user 0.035s sys 0.038s
I20250623 01:58:17.383740 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:17.384788 13807 raft_consensus.cc:2391] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:17.385442 14033 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.385907 14033 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.387874 14033 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.389724 13887 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:17.388922 14266 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.391384 13888 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:17.397861 14004 raft_consensus.cc:357] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.398787 14004 raft_consensus.cc:383] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.399140 14004 raft_consensus.cc:738] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.399356 14104 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.399753 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:17.400496 13956 raft_consensus.cc:2391] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:17.399968 14004 consensus_queue.cc:260] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.408064 14003 tablet_bootstrap.cc:654] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.422793 13926 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:17.423566 14265 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.424742 14004 ts_tablet_manager.cc:1428] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.042s user 0.008s sys 0.001s
I20250623 01:58:17.425628 14004 tablet_bootstrap.cc:492] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:17.437088 13708 catalog_manager.cc:5582] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 2, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:17.449517 13807 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.449599 14002 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:17.450100 14002 ts_tablet_manager.cc:1397] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.266s user 0.018s sys 0.030s
I20250623 01:58:17.458474 14002 raft_consensus.cc:357] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.459105 14002 raft_consensus.cc:383] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.459363 14002 raft_consensus.cc:738] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.461365 14002 consensus_queue.cc:260] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.462287 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.470212 14187 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.471163 14002 ts_tablet_manager.cc:1428] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250623 01:58:17.472069 14002 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:17.474177 14273 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.474661 14273 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.478389 14273 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.478648 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:17.482501 13779 leader_election.cc:343] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.495893 13776 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:17.497582 14271 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:17.498046 14271 raft_consensus.cc:695] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:17.498804 14271 consensus_queue.cc:237] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.498191 14004 tablet_bootstrap.cc:654] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.501267 14033 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.503497 14216 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.504441 14216 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.500963 13776 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:17.506464 14216 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.506980 14273 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.510074 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:17.510885 13959 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:17.512125 13854 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:17.513741 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.514091 14187 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:17.514432 14187 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:17.514781 14187 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.515456 14273 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:17.516324 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:17.519645 13852 leader_election.cc:343] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.521701 14216 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.524200 14265 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.524662 14265 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.526517 14265 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:17.532589 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:17.535220 14002 tablet_bootstrap.cc:654] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:17.537823 13926 leader_election.cc:343] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.539479 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:17.544279 13776 leader_election.cc:343] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
W20250623 01:58:17.558072 13925 leader_election.cc:343] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.562181 13925 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:17.565972 14004 tablet_bootstrap.cc:492] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:17.566483 14265 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.566473 14004 ts_tablet_manager.cc:1397] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.141s user 0.013s sys 0.012s
I20250623 01:58:17.563630 13776 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:17.567778 14271 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.572921 14003 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:17.573498 14003 ts_tablet_manager.cc:1397] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.232s user 0.013s sys 0.006s
W20250623 01:58:17.574491 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:17.584404 13710 catalog_manager.cc:5582] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:17.586313 14187 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.588213 14187 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.588321 14216 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.591434 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:17.595903 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:17.596449 13956 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.598059 14002 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:17.598811 14002 ts_tablet_manager.cc:1397] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.127s user 0.005s sys 0.006s
I20250623 01:58:17.601194 14002 raft_consensus.cc:357] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.601933 14002 raft_consensus.cc:383] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.602234 14002 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.602839 14002 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.604875 14002 ts_tablet_manager.cc:1428] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250623 01:58:17.579418 14003 raft_consensus.cc:357] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.588685 14004 raft_consensus.cc:357] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.605525 14003 raft_consensus.cc:383] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.605597 14004 raft_consensus.cc:383] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.605885 14003 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.605919 14004 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.606555 14003 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.606555 14004 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.608649 14004 ts_tablet_manager.cc:1428] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.041s user 0.003s sys 0.002s
I20250623 01:58:17.608649 14003 ts_tablet_manager.cc:1428] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
W20250623 01:58:17.609190 13852 leader_election.cc:343] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.609383 14002 tablet_bootstrap.cc:492] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:17.610122 14003 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:17.614012 14004 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:17.619382 14004 tablet_bootstrap.cc:654] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.622082 13956 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:17.623682 13854 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:17.624751 14104 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:17.625231 14104 raft_consensus.cc:695] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:17.626140 14002 tablet_bootstrap.cc:654] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.626075 14104 consensus_queue.cc:237] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.633791 14004 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:17.634151 14004 ts_tablet_manager.cc:1397] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.004s
I20250623 01:58:17.641557 14003 tablet_bootstrap.cc:654] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.652966 14004 raft_consensus.cc:357] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.653980 14004 raft_consensus.cc:383] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.654263 14004 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.654932 13710 catalog_manager.cc:5582] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:17.654853 14004 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.673731 14003 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:17.674232 14003 ts_tablet_manager.cc:1397] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.011s
I20250623 01:58:17.676273 14004 ts_tablet_manager.cc:1428] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.042s user 0.007s sys 0.000s
I20250623 01:58:17.678624 14004 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:17.684829 14004 tablet_bootstrap.cc:654] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.676441 14003 raft_consensus.cc:357] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.691599 14003 raft_consensus.cc:383] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.691990 14003 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.692732 14002 tablet_bootstrap.cc:492] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:17.697286 14002 ts_tablet_manager.cc:1397] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.088s user 0.031s sys 0.033s
I20250623 01:58:17.697468 14003 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.699851 14002 raft_consensus.cc:357] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.700588 14002 raft_consensus.cc:383] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.699883 14003 ts_tablet_manager.cc:1428] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250623 01:58:17.700938 14002 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.701602 14002 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.703544 14004 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:17.704372 14004 ts_tablet_manager.cc:1397] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.005s
I20250623 01:58:17.707291 14004 raft_consensus.cc:357] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.703960 14003 tablet_bootstrap.cc:492] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:17.713196 14003 tablet_bootstrap.cc:654] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.713424 14004 raft_consensus.cc:383] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.713747 14004 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.714406 14002 ts_tablet_manager.cc:1428] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20250623 01:58:17.715370 14002 tablet_bootstrap.cc:492] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:17.714334 14004 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.721002 14002 tablet_bootstrap.cc:654] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.737485 14004 ts_tablet_manager.cc:1428] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.033s user 0.004s sys 0.003s
I20250623 01:58:17.740804 14265 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.741271 14265 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.743176 14265 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:17.744496 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.745328 13886 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:17.746500 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:17.747270 13812 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:17.748660 13926 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:17.749286 14004 tablet_bootstrap.cc:492] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:17.750414 14265 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:17.750908 14265 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:17.751318 14265 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.759085 14003 tablet_bootstrap.cc:492] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:17.759665 14003 ts_tablet_manager.cc:1397] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.058s user 0.028s sys 0.020s
I20250623 01:58:17.761186 14265 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.762786 14003 raft_consensus.cc:357] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.763650 14003 raft_consensus.cc:383] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.763685 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:17.764065 14003 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.764374 13812 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.765172 14265 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:17.768528 14004 tablet_bootstrap.cc:654] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.770625 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:17.771164 13886 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.771272 14002 tablet_bootstrap.cc:492] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:17.771759 14002 ts_tablet_manager.cc:1397] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.057s user 0.024s sys 0.019s
I20250623 01:58:17.772526 14003 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.774148 14002 raft_consensus.cc:357] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.774820 14003 ts_tablet_manager.cc:1428] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20250623 01:58:17.774824 14002 raft_consensus.cc:383] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.775339 14002 raft_consensus.cc:738] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.775776 14003 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:17.775996 14002 consensus_queue.cc:260] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.777498 14002 ts_tablet_manager.cc:1428] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20250623 01:58:17.779804 13886 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:17.780982 13925 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:17.781810 14265 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:17.782419 14265 raft_consensus.cc:695] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:17.783183 14265 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.784266 14003 tablet_bootstrap.cc:654] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.790656 13812 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:17.793943 14002 tablet_bootstrap.cc:492] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:17.798283 14003 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:17.798866 14003 ts_tablet_manager.cc:1397] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.007s
I20250623 01:58:17.801505 14002 tablet_bootstrap.cc:654] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.802789 14004 tablet_bootstrap.cc:492] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:17.803283 14004 ts_tablet_manager.cc:1397] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.054s user 0.022s sys 0.009s
I20250623 01:58:17.801589 14003 raft_consensus.cc:357] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.805429 14003 raft_consensus.cc:383] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.805733 14003 raft_consensus.cc:738] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.805822 14004 raft_consensus.cc:357] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.806541 14004 raft_consensus.cc:383] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.806509 14003 consensus_queue.cc:260] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.811290 14004 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.812027 14004 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.814374 14004 ts_tablet_manager.cc:1428] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20250623 01:58:17.815256 14004 tablet_bootstrap.cc:492] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:17.818105 14003 ts_tablet_manager.cc:1428] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250623 01:58:17.821144 14004 tablet_bootstrap.cc:654] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.821499 14266 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.822046 14002 tablet_bootstrap.cc:492] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:17.822537 14002 ts_tablet_manager.cc:1397] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.010s
I20250623 01:58:17.821981 14266 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.825330 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:17.826627 13926 leader_election.cc:343] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.827505 14266 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:17.826653 14002 raft_consensus.cc:357] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.829272 14002 raft_consensus.cc:383] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.829177 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.829674 14002 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.830296 14002 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:17.830646 13925 leader_election.cc:343] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.831174 13925 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:17.832778 14212 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:17.837471 14003 tablet_bootstrap.cc:492] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:17.839643 14002 ts_tablet_manager.cc:1428] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20250623 01:58:17.840648 14002 tablet_bootstrap.cc:492] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:17.840950 14004 tablet_bootstrap.cc:492] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:17.841454 14004 ts_tablet_manager.cc:1397] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.003s
I20250623 01:58:17.843008 13710 catalog_manager.cc:5582] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:17.845633 14004 raft_consensus.cc:357] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.846412 14004 raft_consensus.cc:383] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.846726 14004 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.847345 14004 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.848625 14003 tablet_bootstrap.cc:654] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.850740 14002 tablet_bootstrap.cc:654] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.849519 14004 ts_tablet_manager.cc:1428] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250623 01:58:17.852295 14004 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:17.859931 14004 tablet_bootstrap.cc:654] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.866132 13954 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 01:58:17.867774 14273 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:17.881810 14003 tablet_bootstrap.cc:492] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:17.884649 14003 ts_tablet_manager.cc:1397] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.003s
I20250623 01:58:17.884454 14273 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.894656 13886 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 01:58:17.904927 14273 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.907037 14104 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.907476 14104 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.909076 14104 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.910046 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:17.910629 13954 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:17.911603 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:17.913261 13852 leader_election.cc:343] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.914563 13854 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:17.915334 14104 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:17.915635 14104 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:17.915975 14104 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.914732 14273 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.919384 14273 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.921859 14273 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.922188 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.922515 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:17.923055 14187 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.923534 14187 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.923066 13954 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:17.925283 14187 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.925437 14002 tablet_bootstrap.cc:492] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:17.925362 13779 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:17.926266 14273 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:17.926599 14273 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:17.926904 14273 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:58:17.927367 13776 leader_election.cc:343] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.926777 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:17.927893 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:17.928184 13960 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:17.928767 14104 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.929121 13812 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:17.930634 14104 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.930883 13854 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:17.932123 14003 raft_consensus.cc:357] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.932700 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:17.932955 14003 raft_consensus.cc:383] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.926017 14002 ts_tablet_manager.cc:1397] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.086s user 0.020s sys 0.039s
I20250623 01:58:17.933291 13960 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.937593 14187 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:17.937968 14187 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:17.938266 14187 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.933121 14273 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.944099 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:17.945545 14273 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:17.945710 13852 leader_election.cc:343] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:17.950201 14002 raft_consensus.cc:357] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.950852 14002 raft_consensus.cc:383] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:17.951088 14002 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:17.951659 14002 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.953305 14081 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.953825 14002 ts_tablet_manager.cc:1428] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.020s user 0.004s sys 0.001s
I20250623 01:58:17.953748 14081 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.954717 14002 tablet_bootstrap.cc:492] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:17.955518 14081 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:17.961472 14273 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:17.962323 14265 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:17.962675 14002 tablet_bootstrap.cc:654] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:17.962687 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:17.962792 14265 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:17.965704 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:17.966521 13954 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.969986 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:17.970765 13885 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:17.971661 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:17.979219 13960 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:17.982954 13776 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:18.007109 13776 leader_election.cc:343] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.008517 14269 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.008869 14269 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.009174 14269 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.991081 14265 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:17.992056 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:18.011971 13885 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:17.996788 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:18.012869 13812 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:18.013867 13925 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:18.014782 14212 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.015123 14212 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.015420 14212 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.024749 14212 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.027459 14212 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:18.031313 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:18.032171 13812 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.032351 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:18.034005 13885 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:17.933308 14003 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.038640 14003 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.044986 13807 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.047576 13954 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:18.049293 13779 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:17.987743 14187 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.050627 14081 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.052067 14081 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:18.053031 14081 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.055842 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:18.056483 13807 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:58:18.059942 14304 log.cc:927] Time spent T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20250623 01:58:18.060550 14259 consensus_queue.cc:1035] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:18.063345 14002 tablet_bootstrap.cc:492] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.063779 14002 ts_tablet_manager.cc:1397] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.109s user 0.040s sys 0.047s
I20250623 01:58:18.066679 14002 raft_consensus.cc:357] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.067250 14002 raft_consensus.cc:383] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.067473 14002 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.068013 14002 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.069736 14002 ts_tablet_manager.cc:1428] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250623 01:58:18.070504 14002 tablet_bootstrap.cc:492] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:18.046705 13854 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:18.056586 13885 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:18.021230 14269 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.052083 14187 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.072786 13961 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:18.074818 13925 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:18.081642 13812 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:18.085517 14004 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:18.085940 14004 ts_tablet_manager.cc:1397] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.234s user 0.046s sys 0.067s
I20250623 01:58:18.084918 14216 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.086817 14216 raft_consensus.cc:695] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:18.087634 14216 consensus_queue.cc:237] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.088562 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:18.089116 13885 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.083097 14265 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.091039 14265 raft_consensus.cc:695] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:18.094226 14002 tablet_bootstrap.cc:654] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.100031 14265 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.101145 14269 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:18.095152 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 48526.2us
I20250623 01:58:18.104444 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:18.104971 13961 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.105091 14003 ts_tablet_manager.cc:1428] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.198s user 0.006s sys 0.002s
I20250623 01:58:18.106169 14003 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:18.107296 14215 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.107738 14215 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.112959 13961 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:18.117973 14004 raft_consensus.cc:357] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.120056 14004 raft_consensus.cc:383] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.120494 14004 raft_consensus.cc:738] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.121266 14004 consensus_queue.cc:260] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.108868 13885 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:18.112668 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:18.133692 13779 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:18.136859 14002 tablet_bootstrap.cc:492] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.137414 14002 ts_tablet_manager.cc:1397] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.011s
I20250623 01:58:18.140344 13959 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.140501 14273 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.145895 14273 raft_consensus.cc:695] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:18.146757 14273 consensus_queue.cc:237] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.155469 14004 ts_tablet_manager.cc:1428] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.069s user 0.008s sys 0.000s
I20250623 01:58:18.159493 14004 tablet_bootstrap.cc:492] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:18.161063 14215 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.164582 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.165300 13960 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:18.164551 13807 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:18.172974 13959 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:18.179458 13852 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:18.181123 14057 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.181593 14057 raft_consensus.cc:695] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:18.182406 14057 consensus_queue.cc:237] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.182854 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:18.185962 14002 raft_consensus.cc:357] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.186679 14002 raft_consensus.cc:383] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.186959 14002 raft_consensus.cc:738] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.185495 13708 catalog_manager.cc:5582] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:18.190501 13852 leader_election.cc:343] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.190867 14002 consensus_queue.cc:260] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.195174 13711 catalog_manager.cc:5582] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:18.196607 13710 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:18.199545 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:18.199618 14215 consensus_queue.cc:1035] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.201942 14002 ts_tablet_manager.cc:1428] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.064s user 0.004s sys 0.002s
I20250623 01:58:18.202829 14002 tablet_bootstrap.cc:492] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:18.201278 14215 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.207202 14215 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.207516 14215 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.231029 14265 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.232410 14003 tablet_bootstrap.cc:654] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.232355 14265 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.248944 14002 tablet_bootstrap.cc:654] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.251542 14004 tablet_bootstrap.cc:654] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.258315 14215 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.260071 13708 catalog_manager.cc:5582] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:18.263463 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:18.264775 14215 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.264889 13954 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.265376 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:18.270125 13852 leader_election.cc:343] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.276504 14002 tablet_bootstrap.cc:492] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.279801 14002 ts_tablet_manager.cc:1397] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.009s
I20250623 01:58:18.278818 14269 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.277751 13711 catalog_manager.cc:5582] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:18.280221 14003 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:18.280221 14269 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.280870 14003 ts_tablet_manager.cc:1397] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.175s user 0.015s sys 0.005s
I20250623 01:58:18.282335 14269 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.282917 14002 raft_consensus.cc:357] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.283604 14002 raft_consensus.cc:383] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.283404 14003 raft_consensus.cc:357] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.273942 13954 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:18.284261 14003 raft_consensus.cc:383] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.284404 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.284652 14003 raft_consensus.cc:738] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.285105 13959 raft_consensus.cc:2391] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:18.285283 14003 consensus_queue.cc:260] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.286051 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:18.287099 14215 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.287706 14215 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:18.287854 14003 ts_tablet_manager.cc:1428] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250623 01:58:18.288705 14003 tablet_bootstrap.cc:492] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:18.288478 14215 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.283918 14002 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.285641 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:18.289876 14002 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.290740 13776 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:18.291685 14273 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.292382 14002 ts_tablet_manager.cc:1428] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250623 01:58:18.293361 14002 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:18.301126 14002 tablet_bootstrap.cc:654] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.310484 14273 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:18.314426 14002 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.314821 14002 ts_tablet_manager.cc:1397] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.006s
I20250623 01:58:18.314915 14004 tablet_bootstrap.cc:492] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:18.315351 14004 ts_tablet_manager.cc:1397] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.159s user 0.013s sys 0.000s
I20250623 01:58:18.315480 14003 tablet_bootstrap.cc:654] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.316735 14002 raft_consensus.cc:357] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.317236 14002 raft_consensus.cc:383] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.317534 14002 raft_consensus.cc:738] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.317711 14004 raft_consensus.cc:357] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.318338 14004 raft_consensus.cc:383] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.318738 14004 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.318265 14002 consensus_queue.cc:260] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.323302 14002 ts_tablet_manager.cc:1428] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250623 01:58:18.325224 14002 tablet_bootstrap.cc:492] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:18.328859 14004 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.329562 14265 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.330030 14265 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.331871 14265 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:18.333846 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:18.334493 13885 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:18.335199 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:18.335803 13812 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:18.335842 13925 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:18.336494 13711 catalog_manager.cc:5582] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:18.337349 14265 raft_consensus.cc:2802] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.337733 14265 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.338050 14265 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.346055 14002 tablet_bootstrap.cc:654] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.353122 14003 tablet_bootstrap.cc:492] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:18.353533 14003 ts_tablet_manager.cc:1397] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.004s
I20250623 01:58:18.355615 14003 raft_consensus.cc:357] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.356216 14003 raft_consensus.cc:383] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.356443 14003 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.346843 14004 ts_tablet_manager.cc:1428] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20250623 01:58:18.357486 14004 tablet_bootstrap.cc:492] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:18.359068 14002 tablet_bootstrap.cc:492] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.359493 14002 ts_tablet_manager.cc:1397] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.008s
I20250623 01:58:18.357048 14003 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.361675 14002 raft_consensus.cc:357] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.362286 14002 raft_consensus.cc:383] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.362529 14002 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.362967 14003 ts_tablet_manager.cc:1428] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250623 01:58:18.347585 14265 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.363098 14002 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.367282 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:18.367884 13885 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.368278 14265 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:18.370121 14002 ts_tablet_manager.cc:1428] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250623 01:58:18.371017 14002 tablet_bootstrap.cc:492] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:18.376837 14003 tablet_bootstrap.cc:492] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:18.378545 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:18.379451 13812 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.385080 13885 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:18.393994 13925 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:18.394908 13812 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:18.395998 14265 raft_consensus.cc:2802] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.396709 14002 tablet_bootstrap.cc:654] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.396912 14265 raft_consensus.cc:695] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:18.398267 14265 consensus_queue.cc:237] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.404546 14004 tablet_bootstrap.cc:654] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.410485 13710 catalog_manager.cc:5582] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:18.412127 14003 tablet_bootstrap.cc:654] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.423055 14002 tablet_bootstrap.cc:492] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.423565 14003 tablet_bootstrap.cc:492] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:18.423619 14002 ts_tablet_manager.cc:1397] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.009s
I20250623 01:58:18.425056 14003 ts_tablet_manager.cc:1397] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.008s
I20250623 01:58:18.426702 14002 raft_consensus.cc:357] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.427407 14002 raft_consensus.cc:383] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.427603 14004 tablet_bootstrap.cc:492] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:18.427332 14003 raft_consensus.cc:357] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.427935 14004 ts_tablet_manager.cc:1397] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.000s
I20250623 01:58:18.427750 14002 raft_consensus.cc:738] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.427989 14003 raft_consensus.cc:383] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.428320 14003 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.428967 14003 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.428967 14002 consensus_queue.cc:260] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.429618 14004 raft_consensus.cc:357] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.430187 14004 raft_consensus.cc:383] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.430444 14004 raft_consensus.cc:738] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.431001 14004 consensus_queue.cc:260] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.431416 14002 ts_tablet_manager.cc:1428] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250623 01:58:18.433051 14004 ts_tablet_manager.cc:1428] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250623 01:58:18.433066 14003 ts_tablet_manager.cc:1428] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250623 01:58:18.434177 14004 tablet_bootstrap.cc:492] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:18.434222 14003 tablet_bootstrap.cc:492] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:18.440503 14004 tablet_bootstrap.cc:654] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.440907 14003 tablet_bootstrap.cc:654] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.448067 14002 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:18.455432 14003 tablet_bootstrap.cc:492] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:18.455909 14003 ts_tablet_manager.cc:1397] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20250623 01:58:18.457114 14002 tablet_bootstrap.cc:654] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.462072 14002 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.462494 14002 ts_tablet_manager.cc:1397] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.001s
I20250623 01:58:18.464784 14002 raft_consensus.cc:357] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.465842 14002 raft_consensus.cc:383] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.466111 14002 raft_consensus.cc:738] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.466655 14002 consensus_queue.cc:260] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.458379 14003 raft_consensus.cc:357] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.468987 14003 raft_consensus.cc:383] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.469089 14002 ts_tablet_manager.cc:1428] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250623 01:58:18.469260 14003 raft_consensus.cc:738] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.469549 14004 tablet_bootstrap.cc:492] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:18.469967 14004 ts_tablet_manager.cc:1397] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.007s
I20250623 01:58:18.469987 14002 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:18.469879 14003 consensus_queue.cc:260] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.470906 14273 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.471390 14273 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.472070 14003 ts_tablet_manager.cc:1428] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20250623 01:58:18.472435 14004 raft_consensus.cc:357] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.473536 14003 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:18.473625 14004 raft_consensus.cc:383] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.473966 14004 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.474503 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:18.475113 13885 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:18.475505 14273 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.476112 13776 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:18.476883 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.477460 13959 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:18.481284 14033 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.481645 14033 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.481906 14033 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.482286 14004 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.483189 14002 tablet_bootstrap.cc:654] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.483116 14081 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.483632 14081 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.484328 14004 ts_tablet_manager.cc:1428] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20250623 01:58:18.486146 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:18.490787 13776 leader_election.cc:343] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.491333 14081 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.492871 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.493463 13959 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:18.494334 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:18.495208 14081 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.495536 14081 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.495823 14081 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.497087 14003 tablet_bootstrap.cc:654] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.500010 14033 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.485112 14004 tablet_bootstrap.cc:492] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:18.519104 14033 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.515623 13885 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.517048 14271 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.525543 14271 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.527335 14271 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.519994 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:18.529067 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:18.529647 13888 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.530607 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.531509 13954 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:18.521358 14003 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:18.523172 14334 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.532716 14334 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.535261 14334 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.535774 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:18.535961 13888 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:18.503158 14081 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:18.537242 13776 leader_election.cc:343] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.537940 13776 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:18.536060 14003 ts_tablet_manager.cc:1397] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.001s
I20250623 01:58:18.538676 14334 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.539110 14334 raft_consensus.cc:695] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:18.539505 14081 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.539875 14334 consensus_queue.cc:237] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.541468 14003 raft_consensus.cc:357] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.542194 14003 raft_consensus.cc:383] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.542418 14215 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.542472 14003 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.542979 14215 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.536707 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:18.544138 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:18.544376 13959 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.524379 14004 tablet_bootstrap.cc:654] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:18.546079 13776 leader_election.cc:343] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.546144 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.548341 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:18.548542 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.549083 13956 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
W20250623 01:58:18.549748 13852 leader_election.cc:343] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.550383 13854 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:18.551101 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:18.551156 14187 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.551525 13954 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.544981 14215 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.552585 13779 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:18.553615 13779 leader_election.cc:343] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.554037 14311 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.554453 14311 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.554507 14033 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.554005 13779 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:18.557575 13959 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:18.557675 13954 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:18.543267 14003 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.561892 14081 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250623 01:58:18.564991 14271 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:18.562270 14311 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.573297 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:18.572888 14033 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:18.573891 14081 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.574352 14081 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:18.575268 14081 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.580265 14003 ts_tablet_manager.cc:1428] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20250623 01:58:18.581167 14003 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:18.592988 14002 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.593596 14002 ts_tablet_manager.cc:1397] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.124s user 0.019s sys 0.039s
I20250623 01:58:18.598500 14002 raft_consensus.cc:357] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.599364 14002 raft_consensus.cc:383] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.599694 14002 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.601814 14002 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.611176 14002 ts_tablet_manager.cc:1428] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250623 01:58:18.612195 14002 tablet_bootstrap.cc:492] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:18.619020 14215 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.620453 14311 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.631311 14271 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.654124 14004 tablet_bootstrap.cc:492] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:18.654565 14004 ts_tablet_manager.cc:1397] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.170s user 0.003s sys 0.018s
I20250623 01:58:18.659855 13710 catalog_manager.cc:5582] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:18.663293 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:18.664678 13852 leader_election.cc:343] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.665364 14311 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.665444 14057 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:18.669728 14003 tablet_bootstrap.cc:654] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.671371 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:18.671973 13956 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.674140 14004 raft_consensus.cc:357] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.674727 14004 raft_consensus.cc:383] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.674952 14004 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.675489 14004 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.678515 14002 tablet_bootstrap.cc:654] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:18.679193 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:18.681123 14004 ts_tablet_manager.cc:1428] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20250623 01:58:18.681984 14004 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:18.685608 14212 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 01:58:18.695825 14004 tablet_bootstrap.cc:654] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:18.700613 14265 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.704497 14033 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.708345 14057 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:18.730089 14081 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.730669 14081 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.734324 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:18.735004 13888 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:18.736109 13776 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:18.737012 14271 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.738767 14057 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:18.739168 14271 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.740684 14271 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.740813 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.741477 13959 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:18.742763 14081 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.753423 14273 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:18.759889 13956 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:18.762555 14004 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:18.770475 14004 ts_tablet_manager.cc:1397] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.007s
I20250623 01:58:18.772104 13711 catalog_manager.cc:5582] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:18.766983 14271 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.790243 13854 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:18.794356 14215 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:18.795001 14215 raft_consensus.cc:695] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:18.796046 14215 consensus_queue.cc:237] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.798035 14081 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.798643 14266 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:18.798691 14081 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.802547 14081 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.804529 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:18.807479 13776 leader_election.cc:343] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:18.808858 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.809581 13954 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:18.810554 14216 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:18.812417 14334 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:18.812999 14334 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.787662 14004 raft_consensus.cc:357] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.818945 14004 raft_consensus.cc:383] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.819278 14004 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:18.820869 13779 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:18.823752 14081 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:18.824162 14081 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:18.824882 14081 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:18.822839 14004 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.821547 14334 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.837780 14004 ts_tablet_manager.cc:1428] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20250623 01:58:18.840935 14004 tablet_bootstrap.cc:492] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:18.844470 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:18.845134 13885 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:18.846344 14002 tablet_bootstrap.cc:492] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:18.846786 14002 ts_tablet_manager.cc:1397] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.235s user 0.026s sys 0.043s
I20250623 01:58:18.871192 14081 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.942912 14003 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:18.943514 14003 ts_tablet_manager.cc:1397] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.363s user 0.032s sys 0.047s
I20250623 01:58:18.972905 14081 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:18.978193 14002 raft_consensus.cc:357] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.978899 14002 raft_consensus.cc:383] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:18.979148 14002 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
W20250623 01:58:18.942065 14350 log.cc:927] Time spent T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
W20250623 01:58:18.980201 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 109318us
I20250623 01:58:18.980476 14002 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:18.991336 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:18.992200 13959 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:19.009747 14002 ts_tablet_manager.cc:1428] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.163s user 0.003s sys 0.005s
I20250623 01:58:19.011198 14002 tablet_bootstrap.cc:492] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:19.021281 14003 raft_consensus.cc:357] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.026940 14003 raft_consensus.cc:383] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:19.028597 14003 raft_consensus.cc:738] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:19.036216 14003 consensus_queue.cc:260] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:19.037716 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 99219.2us
I20250623 01:58:19.040645 13779 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:19.046092 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:19.046614 13954 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:19.056877 14003 ts_tablet_manager.cc:1428] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.113s user 0.009s sys 0.001s
I20250623 01:58:19.059969 14081 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:19.060515 14081 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:19.060892 14081 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:19.064117 14003 tablet_bootstrap.cc:492] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:19.037259 14271 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:19.089339 14081 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.117611 14081 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:19.054562 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:19.136029 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 136586us
I20250623 01:58:19.124387 13959 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:19.156549 13954 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:19.141644 14212 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:19.173019 13779 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:19.178143 14033 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:19.179656 14033 raft_consensus.cc:695] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:19.180517 14033 consensus_queue.cc:237] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.183037 14002 tablet_bootstrap.cc:654] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:19.167418 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:19.198299 14352 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250623 01:58:19.224695 14004 tablet_bootstrap.cc:654] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:19.178520 14212 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.203199 13962 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:58:19.405552 14363 log.cc:927] Time spent T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20250623 01:58:19.446369 13708 catalog_manager.cc:5582] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:19.510214 14002 tablet_bootstrap.cc:492] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:19.481225 13959 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:19.522267 14002 ts_tablet_manager.cc:1397] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.511s user 0.009s sys 0.002s
I20250623 01:58:19.536159 14003 tablet_bootstrap.cc:654] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:19.547947 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 44966.2us
I20250623 01:58:19.555567 14004 tablet_bootstrap.cc:492] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:19.555965 14004 ts_tablet_manager.cc:1397] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.717s user 0.015s sys 0.011s
I20250623 01:58:19.506732 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:19.563812 13812 raft_consensus.cc:2466] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:19.558450 14004 raft_consensus.cc:357] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.571966 14004 raft_consensus.cc:383] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:19.572132 13779 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:19.571977 14002 raft_consensus.cc:357] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.572795 14004 raft_consensus.cc:738] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:19.572930 14002 raft_consensus.cc:383] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:19.573477 14004 consensus_queue.cc:260] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.521245 13962 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:19.576900 14081 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:19.554479 14368 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:19.601346 14368 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.603556 14081 raft_consensus.cc:695] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:19.604322 14081 consensus_queue.cc:237] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.606122 14368 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:19.573403 14002 raft_consensus.cc:738] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:19.608913 14002 consensus_queue.cc:260] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.612843 14003 tablet_bootstrap.cc:492] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:19.613487 14003 ts_tablet_manager.cc:1397] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.556s user 0.041s sys 0.022s
I20250623 01:58:19.517710 14212 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:19.621651 14004 ts_tablet_manager.cc:1428] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.065s user 0.004s sys 0.003s
I20250623 01:58:19.622534 14004 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:19.631760 14003 raft_consensus.cc:357] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.632475 14003 raft_consensus.cc:383] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:19.632740 14003 raft_consensus.cc:738] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:19.574992 14271 consensus_queue.cc:1035] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20250623 01:58:19.633342 14003 consensus_queue.cc:260] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.635730 14003 ts_tablet_manager.cc:1428] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250623 01:58:19.636181 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:19.636936 13886 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:19.638461 14003 tablet_bootstrap.cc:492] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:19.639719 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:19.629745 14368 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20250623 01:58:19.645421 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:19.646054 13882 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:19.646886 14002 ts_tablet_manager.cc:1428] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.124s user 0.013s sys 0.010s
I20250623 01:58:19.647673 14002 tablet_bootstrap.cc:492] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:19.655232 14359 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:19.659886 13779 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:19.661589 13926 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:19.667945 14334 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:19.668092 14372 raft_consensus.cc:2802] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:19.669559 14372 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:19.670722 14372 raft_consensus.cc:3058] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:19.668774 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:19.668690 14334 raft_consensus.cc:695] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:19.692772 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:19.696250 14334 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.709767 14081 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250623 01:58:19.726428 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:19.700533 13886 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
W20250623 01:58:19.727495 13925 leader_election.cc:343] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:19.741935 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:19.754598 14372 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.768687 14212 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250623 01:58:19.779641 14003 tablet_bootstrap.cc:654] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:19.785566 14004 tablet_bootstrap.cc:654] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:19.794371 13808 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:19.803848 14266 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:19.804533 14266 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:19.807032 14266 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:19.810987 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:19.826223 13776 leader_election.cc:343] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
W20250623 01:58:19.825858 13926 leader_election.cc:343] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:19.846284 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:19.852267 13925 leader_election.cc:343] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:19.852725 13925 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:19.862968 14365 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:19.867439 13882 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
W20250623 01:58:19.872318 13776 consensus_peers.cc:487] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Couldn't send request to peer ee041a8325864e788a383b8356afc652. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:19.879228 14002 tablet_bootstrap.cc:654] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:19.885413 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:19.886267 13806 raft_consensus.cc:3058] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:19.904793 14081 consensus_queue.cc:1035] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:19.926054 14372 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:19.974725 14388 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250623 01:58:19.981094 13710 catalog_manager.cc:5582] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:19.983410 14369 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:19.991290 14004 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:19.992611 14004 ts_tablet_manager.cc:1397] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.370s user 0.024s sys 0.016s
I20250623 01:58:19.991626 13776 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:19.999085 14273 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:19.935544 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:19.993721 14369 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:20.014411 13925 leader_election.cc:343] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.025913 14368 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 01:58:20.030599 14004 raft_consensus.cc:357] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.031530 14004 raft_consensus.cc:383] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.031837 14004 raft_consensus.cc:738] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.038147 14369 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.041467 14002 tablet_bootstrap.cc:492] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:20.041946 14002 ts_tablet_manager.cc:1397] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.394s user 0.016s sys 0.019s
I20250623 01:58:20.041500 14003 tablet_bootstrap.cc:492] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:20.042788 14003 ts_tablet_manager.cc:1397] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.405s user 0.017s sys 0.008s
W20250623 01:58:20.044518 14391 log.cc:927] Time spent T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20250623 01:58:20.041224 14004 consensus_queue.cc:260] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.045097 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:20.048347 13854 leader_election.cc:343] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.062084 14004 ts_tablet_manager.cc:1428] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.069s user 0.006s sys 0.001s
I20250623 01:58:20.063102 14004 tablet_bootstrap.cc:492] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:20.051290 14369 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:20.069597 14377 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.070138 14377 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.073037 14377 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.081043 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:20.081696 13888 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:20.087888 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.089428 14273 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:20.090119 14273 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.091390 14004 tablet_bootstrap.cc:654] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.094624 14273 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.096635 13806 raft_consensus.cc:2466] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:20.098623 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.100049 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.100690 13810 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:20.101631 14002 raft_consensus.cc:357] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.102306 14002 raft_consensus.cc:383] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.102556 14002 raft_consensus.cc:738] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.103210 13776 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:20.104058 14377 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:20.104187 13852 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:20.104424 14377 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:20.103128 14002 consensus_queue.cc:260] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.104797 14377 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.106024 14002 ts_tablet_manager.cc:1428] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
I20250623 01:58:20.106917 14002 tablet_bootstrap.cc:492] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
W20250623 01:58:20.106998 13779 leader_election.cc:343] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.108204 13926 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:20.109689 14003 raft_consensus.cc:357] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.110271 14372 raft_consensus.cc:2802] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:20.110368 14003 raft_consensus.cc:383] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.110807 14360 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:20.111261 14360 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:20.110760 14372 raft_consensus.cc:695] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:20.115216 14355 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.115685 14355 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.110765 14003 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.112824 14360 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.118362 14003 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.120657 14003 ts_tablet_manager.cc:1428] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.078s user 0.002s sys 0.004s
I20250623 01:58:20.123282 14003 tablet_bootstrap.cc:492] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:20.124658 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:20.141239 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.135193 14372 consensus_queue.cc:237] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.141597 13776 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:20.143597 13880 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.148620 14357 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:20.145861 14355 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.163630 14033 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:20.165724 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.166556 13962 raft_consensus.cc:2391] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:20.168457 13854 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:20.181932 14273 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:20.189226 14357 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.192380 13710 catalog_manager.cc:5582] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:20.199385 13710 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:20.190927 14360 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.198697 14357 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.209383 14357 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.210755 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.213768 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:20.214459 13779 leader_election.cc:343] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
W20250623 01:58:20.216399 13776 leader_election.cc:343] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.216988 13776 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:20.218182 14334 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:20.220311 14377 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.226709 14311 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:20.232437 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:20.234026 14004 tablet_bootstrap.cc:492] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:20.234637 14004 ts_tablet_manager.cc:1397] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.172s user 0.016s sys 0.006s
I20250623 01:58:20.237353 14360 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.240097 14002 tablet_bootstrap.cc:654] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.240434 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:20.241259 13808 raft_consensus.cc:2391] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:20.242941 14003 tablet_bootstrap.cc:654] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
W20250623 01:58:20.244499 13779 leader_election.cc:343] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.245983 14033 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:20.250771 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:20.251818 14368 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:20.252471 13888 raft_consensus.cc:2391] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:20.253703 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:20.247478 13711 catalog_manager.cc:5582] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:20.243985 14377 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:20.273591 13854 leader_election.cc:343] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.276180 13854 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:20.277659 13776 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:20.278515 14273 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 01:58:20.281929 14311 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 01:58:20.280733 14004 raft_consensus.cc:357] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.283488 14004 raft_consensus.cc:383] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.283715 14004 raft_consensus.cc:738] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.284677 14004 consensus_queue.cc:260] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.291110 14004 ts_tablet_manager.cc:1428] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.056s user 0.005s sys 0.000s
I20250623 01:58:20.292994 14004 tablet_bootstrap.cc:492] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:20.296540 14003 tablet_bootstrap.cc:492] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:20.297060 14003 ts_tablet_manager.cc:1397] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.174s user 0.008s sys 0.004s
I20250623 01:58:20.310153 14004 tablet_bootstrap.cc:654] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.312237 14002 tablet_bootstrap.cc:492] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:20.312713 14002 ts_tablet_manager.cc:1397] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.206s user 0.010s sys 0.010s
I20250623 01:58:20.308004 14003 raft_consensus.cc:357] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.316823 14002 raft_consensus.cc:357] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.314519 14003 raft_consensus.cc:383] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.317534 14002 raft_consensus.cc:383] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.317677 14003 raft_consensus.cc:738] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.317852 14002 raft_consensus.cc:738] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.318490 14003 consensus_queue.cc:260] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.318490 14002 consensus_queue.cc:260] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.322876 14002 ts_tablet_manager.cc:1428] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250623 01:58:20.324702 14002 tablet_bootstrap.cc:492] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:20.336326 14002 tablet_bootstrap.cc:654] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.348127 14003 ts_tablet_manager.cc:1428] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.051s user 0.004s sys 0.003s
I20250623 01:58:20.348946 14002 tablet_bootstrap.cc:492] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:20.349424 14002 ts_tablet_manager.cc:1397] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20250623 01:58:20.354626 14003 tablet_bootstrap.cc:492] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:20.368608 14377 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.364622 14365 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.381065 14365 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.383518 14365 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:20.383647 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.384186 14377 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:20.385190 13926 leader_election.cc:343] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.386147 14003 tablet_bootstrap.cc:654] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.386073 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:20.387562 13925 leader_election.cc:343] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.387933 13925 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:20.398315 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:20.400663 14377 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.354880 14002 raft_consensus.cc:357] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.401930 14002 raft_consensus.cc:383] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.402215 14002 raft_consensus.cc:738] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.402809 14002 consensus_queue.cc:260] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.404769 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.407966 14365 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 01:58:20.408876 13776 leader_election.cc:343] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
W20250623 01:58:20.410049 13779 leader_election.cc:343] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.410421 13779 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:20.411402 14273 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:20.413822 14002 ts_tablet_manager.cc:1428] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.064s user 0.005s sys 0.006s
I20250623 01:58:20.415555 14002 tablet_bootstrap.cc:492] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:20.416199 14004 tablet_bootstrap.cc:492] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:20.416637 14004 ts_tablet_manager.cc:1397] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.124s user 0.024s sys 0.010s
I20250623 01:58:20.418998 14004 raft_consensus.cc:357] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.419660 14004 raft_consensus.cc:383] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.419898 14004 raft_consensus.cc:738] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.420904 14004 consensus_queue.cc:260] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.422056 14002 tablet_bootstrap.cc:654] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.426841 14003 tablet_bootstrap.cc:492] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:20.427302 14003 ts_tablet_manager.cc:1397] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.073s user 0.022s sys 0.015s
I20250623 01:58:20.429714 14003 raft_consensus.cc:357] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.430325 14003 raft_consensus.cc:383] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.430580 14003 raft_consensus.cc:738] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.437799 14003 consensus_queue.cc:260] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.439417 14003 ts_tablet_manager.cc:1428] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20250623 01:58:20.432152 14004 ts_tablet_manager.cc:1428] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250623 01:58:20.441184 14003 tablet_bootstrap.cc:492] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:20.445478 14004 tablet_bootstrap.cc:492] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:20.449676 14004 tablet_bootstrap.cc:654] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.452023 14002 tablet_bootstrap.cc:492] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:20.452531 14002 ts_tablet_manager.cc:1397] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.001s
I20250623 01:58:20.455717 14002 raft_consensus.cc:357] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.456615 14002 raft_consensus.cc:383] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.456938 14002 raft_consensus.cc:738] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.460072 14003 tablet_bootstrap.cc:654] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.464171 14002 consensus_queue.cc:260] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.470634 14002 ts_tablet_manager.cc:1428] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s
I20250623 01:58:20.472795 14002 tablet_bootstrap.cc:492] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:20.475345 14004 tablet_bootstrap.cc:492] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:20.475801 14004 ts_tablet_manager.cc:1397] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.013s
I20250623 01:58:20.478169 14004 raft_consensus.cc:357] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.478807 14004 raft_consensus.cc:383] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.479063 14004 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.484860 14002 tablet_bootstrap.cc:654] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.485010 14004 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.487516 14003 tablet_bootstrap.cc:492] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:20.488359 14003 ts_tablet_manager.cc:1397] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.006s
I20250623 01:58:20.491245 14355 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.491308 14003 raft_consensus.cc:357] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.492056 14003 raft_consensus.cc:383] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.491794 14355 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.492324 14003 raft_consensus.cc:738] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.492522 14004 ts_tablet_manager.cc:1428] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250623 01:58:20.492980 14003 consensus_queue.cc:260] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.493714 14355 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.495479 14003 ts_tablet_manager.cc:1428] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250623 01:58:20.495896 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.496529 13958 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:20.498180 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.498381 13854 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:20.499224 14355 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:20.499581 14355 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:20.499910 14355 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:58:20.501878 13852 leader_election.cc:343] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.503460 14004 tablet_bootstrap.cc:492] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:20.507737 14355 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.509649 14004 tablet_bootstrap.cc:654] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.510756 14003 tablet_bootstrap.cc:492] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:20.511384 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:20.517108 13852 leader_election.cc:343] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.522594 14002 tablet_bootstrap.cc:492] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:20.523087 14002 ts_tablet_manager.cc:1397] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20250623 01:58:20.525772 14002 raft_consensus.cc:357] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.526424 14002 raft_consensus.cc:383] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.526679 14002 raft_consensus.cc:738] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.527310 14002 consensus_queue.cc:260] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.529338 14002 ts_tablet_manager.cc:1428] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250623 01:58:20.530207 14002 tablet_bootstrap.cc:492] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382: Bootstrap starting.
I20250623 01:58:20.534663 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:20.535264 13958 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.536338 14002 tablet_bootstrap.cc:654] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.541337 14004 tablet_bootstrap.cc:492] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:20.541790 14004 ts_tablet_manager.cc:1397] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.018s
I20250623 01:58:20.542418 13958 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:20.543498 13854 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:20.545725 14360 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:20.546177 14360 raft_consensus.cc:695] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:20.546627 14355 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.546993 14360 consensus_queue.cc:237] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.552671 14004 raft_consensus.cc:357] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.553378 14004 raft_consensus.cc:383] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.553651 14004 raft_consensus.cc:738] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.554287 14004 consensus_queue.cc:260] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.557813 14004 ts_tablet_manager.cc:1428] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250623 01:58:20.561148 14003 tablet_bootstrap.cc:654] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.562067 14002 tablet_bootstrap.cc:492] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382: No bootstrap required, opened a new log
I20250623 01:58:20.562559 14002 ts_tablet_manager.cc:1397] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.009s
I20250623 01:58:20.564147 14004 tablet_bootstrap.cc:492] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:20.566520 14360 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.567025 14360 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.569116 14002 raft_consensus.cc:357] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.569873 14002 raft_consensus.cc:383] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.570130 14002 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.572887 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.573282 14002 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.573773 13958 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:20.576483 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.577198 13808 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:20.578727 13852 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:20.581003 14002 ts_tablet_manager.cc:1428] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250623 01:58:20.582118 14004 tablet_bootstrap.cc:654] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.586432 14355 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:20.586848 14355 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:20.587162 14355 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.601086 14360 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.614686 14355 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.618579 14355 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.619398 14003 tablet_bootstrap.cc:492] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:20.619882 14003 ts_tablet_manager.cc:1397] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.109s user 0.016s sys 0.010s
I20250623 01:58:20.620760 14004 tablet_bootstrap.cc:492] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:20.621235 14004 ts_tablet_manager.cc:1397] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.016s
I20250623 01:58:20.622309 14003 raft_consensus.cc:357] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.622987 14003 raft_consensus.cc:383] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.623235 14003 raft_consensus.cc:738] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.623895 14003 consensus_queue.cc:260] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.625034 14004 raft_consensus.cc:357] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.625771 14004 raft_consensus.cc:383] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.626049 14004 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.626698 14004 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.629062 14004 ts_tablet_manager.cc:1428] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250623 01:58:20.630028 14004 tablet_bootstrap.cc:492] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:20.631251 13708 catalog_manager.cc:5582] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:20.636073 14004 tablet_bootstrap.cc:654] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.637650 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:20.638317 13810 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.639019 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:20.639568 13958 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.642427 14003 ts_tablet_manager.cc:1428] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20250623 01:58:20.652952 14003 tablet_bootstrap.cc:492] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635: Bootstrap starting.
I20250623 01:58:20.663558 13810 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:20.664534 14003 tablet_bootstrap.cc:654] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.667223 13852 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:20.668520 14187 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:20.668995 14187 raft_consensus.cc:695] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:20.670284 14187 consensus_queue.cc:237] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.690299 13958 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:20.695968 14004 tablet_bootstrap.cc:492] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:20.696406 14004 ts_tablet_manager.cc:1397] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.067s user 0.014s sys 0.008s
I20250623 01:58:20.698770 14004 raft_consensus.cc:357] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.699397 14004 raft_consensus.cc:383] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.699659 14004 raft_consensus.cc:738] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.700999 14004 consensus_queue.cc:260] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.703032 14004 ts_tablet_manager.cc:1428] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250623 01:58:20.703836 14004 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:20.705176 13711 catalog_manager.cc:5582] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:20.705880 14360 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.706384 14360 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.708257 14360 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.709364 14004 tablet_bootstrap.cc:654] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.709236 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.711431 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.712808 13852 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:20.714056 14360 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 01:58:20.721663 13925 consensus_peers.cc:487] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 -> Peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Couldn't send request to peer ee041a8325864e788a383b8356afc652. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:20.728734 14365 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:20.764889 14368 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.765386 14368 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.767284 14368 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.769152 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.769827 13956 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:20.770183 14003 tablet_bootstrap.cc:492] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635: No bootstrap required, opened a new log
I20250623 01:58:20.770759 14003 ts_tablet_manager.cc:1397] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635: Time spent bootstrapping tablet: real 0.118s user 0.008s sys 0.014s
I20250623 01:58:20.771863 13779 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:20.772634 14033 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:20.773077 14033 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:20.773605 14033 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.781594 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:20.782274 13888 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 0.
I20250623 01:58:20.783430 14003 raft_consensus.cc:357] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.784099 14003 raft_consensus.cc:383] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.784368 14003 raft_consensus.cc:738] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.784981 14003 consensus_queue.cc:260] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.787132 14003 ts_tablet_manager.cc:1428] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20250623 01:58:20.791332 14004 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:20.791828 14004 ts_tablet_manager.cc:1397] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.088s user 0.027s sys 0.029s
I20250623 01:58:20.793766 14033 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.801683 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:20.802295 13888 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.806071 14187 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.806562 14187 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.808688 14033 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.814605 14187 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.819154 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.820094 13956 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:20.821259 14365 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.821965 13854 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:20.823206 14360 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:20.823561 14360 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:20.823904 14360 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.825589 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.826253 13806 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 0.
I20250623 01:58:20.825735 13888 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:20.821795 14365 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.830071 13776 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:20.829277 14365 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:20.831059 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:20.831331 14273 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:20.831831 14273 raft_consensus.cc:695] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:20.832017 14004 raft_consensus.cc:357] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.832669 14004 raft_consensus.cc:383] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.832535 14273 consensus_queue.cc:237] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.833979 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:20.832934 14004 raft_consensus.cc:738] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.834558 13958 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.834585 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:20.834916 14004 consensus_queue.cc:260] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:20.835986 13925 leader_election.cc:343] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.840970 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.842073 13806 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:20.842584 14372 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.842765 14360 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.843079 14372 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.844455 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:20.845230 14366 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:20.845667 14366 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:20.845503 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:20.845818 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:20.846050 14366 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.846288 13806 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.846340 13956 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.846789 14360 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.842202 14004 ts_tablet_manager.cc:1428] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.050s user 0.002s sys 0.005s
I20250623 01:58:20.847992 14004 tablet_bootstrap.cc:492] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652: Bootstrap starting.
I20250623 01:58:20.846378 14372 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:20.851867 13958 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:20.852726 14366 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.854300 13956 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:20.855192 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:20.855589 14004 tablet_bootstrap.cc:654] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:20.855988 13888 raft_consensus.cc:2391] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:20.857223 14366 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:20.858242 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:20.858532 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.860533 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:20.861093 13808 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.861287 13810 raft_consensus.cc:2391] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 1: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
W20250623 01:58:20.861349 13925 leader_election.cc:343] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Illegal state: must be running to vote when last-logged opid is not known
I20250623 01:58:20.862985 13854 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:20.864050 14360 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:20.864498 14360 raft_consensus.cc:695] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:20.865283 14360 consensus_queue.cc:237] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.868788 13926 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:20.871004 13808 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:20.871165 14366 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.874305 14004 tablet_bootstrap.cc:492] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652: No bootstrap required, opened a new log
I20250623 01:58:20.874845 14004 ts_tablet_manager.cc:1397] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.004s
I20250623 01:58:20.875105 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:20.877220 14004 raft_consensus.cc:357] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.877863 14372 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:20.878079 14004 raft_consensus.cc:383] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:20.878510 14004 raft_consensus.cc:738] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Initialized, Role: FOLLOWER
I20250623 01:58:20.878441 14372 raft_consensus.cc:695] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:20.879693 14372 consensus_queue.cc:237] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.880239 14366 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:20.879700 14004 consensus_queue.cc:260] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.885053 14004 ts_tablet_manager.cc:1428] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250623 01:58:20.886600 13806 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:20.901240 13708 catalog_manager.cc:5582] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:20.909673 13711 catalog_manager.cc:5582] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:20.913818 13710 catalog_manager.cc:5582] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:20.922796 14360 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.923319 14360 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.925765 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.925200 14360 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.927613 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.928575 13854 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:20.929124 14360 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:20.929376 14369 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:20.929808 14369 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:20.931107 14369 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:20.931819 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:20.931932 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:20.932428 13806 raft_consensus.cc:2391] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 1: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:20.933197 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:20.933751 14369 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:20.939597 14369 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:20.980159 13888 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:21.003127 14273 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:21.003623 14273 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:21.005057 14273 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:21.005620 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:21.005931 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:21.006651 13958 raft_consensus.cc:2391] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 1: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:21.007360 13779 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:21.008035 14033 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:21.013105 14033 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:58:21.046680 14372 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:21.047173 14372 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:21.049068 14372 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:21.050035 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:21.050372 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:21.050608 13888 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:21.050943 13806 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 0.
I20250623 01:58:21.051738 13925 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:21.052397 14366 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:21.052763 14366 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:21.053025 14366 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:21.058389 14366 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:21.060613 14366 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:21.060729 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:21.061163 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:21.061281 13808 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:21.061653 13888 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:21.065977 13888 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:21.066067 13808 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:21.066916 13925 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:21.067485 14366 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:21.067866 14366 raft_consensus.cc:695] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:21.068622 14366 consensus_queue.cc:237] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:21.078325 13711 catalog_manager.cc:5582] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 0 to 1, leader changed from <none> to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:21.198998 14359 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.224223 14057 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 01:58:21.228606 14359 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.280326 14273 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.288075 14434 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.303587 13882 raft_consensus.cc:3058] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:21.332474 14434 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.342135 14187 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.411552 14365 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.418431 14368 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.480332 14366 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 01:58:21.536069 14366 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:21.547693 14372 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250623 01:58:21.864770 13759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250623 01:58:22.316212 14432 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:22.316783 14432 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:22.318805 14432 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:22.319635 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:22.319778 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:22.320453 13810 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:22.320848 13958 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:22.321807 13852 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:22.322646 14432 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:22.323006 14432 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:22.323334 14432 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:22.330727 14432 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:22.332144 14432 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:22.333022 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:22.333549 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:22.333588 13958 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:22.334023 13810 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:22.338995 13958 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:22.339972 13854 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:22.339865 13810 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:22.341243 14432 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:22.341928 14432 raft_consensus.cc:695] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:22.343449 14432 consensus_queue.cc:237] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:22.352602 13710 catalog_manager.cc:5582] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 0 to 2, leader changed from <none> to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:22.923574 13807 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 01:58:22.924960 14432 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:22.928068 13960 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 01:58:22.930755 14432 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:23.835440 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:23.851311 14469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:23.851328 14472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:23.852128 14470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:23.853289 12636 server_base.cc:1048] running on GCE node
I20250623 01:58:23.854933 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:58:23.855163 12636 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:58:23.855329 12636 hybrid_clock.cc:648] HybridClock initialized: now 1750643903855304 us; error 0 us; skew 500 ppm
I20250623 01:58:23.855839 12636 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:23.863116 12636 webserver.cc:469] Webserver started at http://127.12.87.4:33287/ using document root <none> and password file <none>
I20250623 01:58:23.863692 12636 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:23.863931 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:23.864321 12636 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:23.865705 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-3-root/instance:
uuid: "7d22741cfb9b408684cb82c58241b2f5"
format_stamp: "Formatted at 2025-06-23 01:58:23 on dist-test-slave-47g2"
I20250623 01:58:23.871136 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250623 01:58:23.876417 14477 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:23.877216 12636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250623 01:58:23.877532 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-3-root
uuid: "7d22741cfb9b408684cb82c58241b2f5"
format_stamp: "Formatted at 2025-06-23 01:58:23 on dist-test-slave-47g2"
I20250623 01:58:23.877782 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:23.896696 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:23.898140 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:23.900197 12636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:23.902659 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:23.902920 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:23.903219 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:23.903393 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:24.030284 12636 rpc_server.cc:307] RPC server started. Bound to: 127.12.87.4:39393
I20250623 01:58:24.030371 14540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.87.4:39393 every 8 connection(s)
I20250623 01:58:24.035455 12636 auto_leader_rebalancer-test.cc:417] add a tserver: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:24.047899 14541 heartbeater.cc:344] Connected to a master server at 127.12.87.62:45405
I20250623 01:58:24.048341 14541 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:24.049216 14541 heartbeater.cc:507] Master 127.12.87.62:45405 requested a full tablet report, sending...
I20250623 01:58:24.051016 13710 ts_manager.cc:194] Registered new tserver with Master: 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:24.052255 13710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38034
I20250623 01:58:25.054515 14541 heartbeater.cc:499] Master 127.12.87.62:45405 was elected leader, sending a full tablet report...
I20250623 01:58:25.077303 13880 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.083076 13960 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:25.084450 13807 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:25.086410 14547 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:25.094959 14359 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:25.107223 14359 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.115887 13958 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
W20250623 01:58:25.120702 13853 consensus_peers.cc:487] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cb699d5420143efa03140964d5aa739. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:25.125933 13807 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.134092 13708 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.157101 13807 consensus_queue.cc:237] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.173731 13958 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:25.174939 13888 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:25.175048 14357 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:25.177881 14273 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250623 01:58:25.189132 13778 consensus_peers.cc:487] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2540092c63cb4d0cbe07ae4018f99f46. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:25.194820 14357 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.197079 13958 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.197806 13888 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.207782 13880 consensus_queue.cc:237] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.212759 13711 catalog_manager.cc:5582] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
W20250623 01:58:25.221071 13853 consensus_peers.cc:487] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 433978dce7d94bcb8968bc09583357c4. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:25.221868 13958 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:25.222913 13807 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:25.223475 14547 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:25.224948 14545 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:25.234740 14553 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.238031 13958 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.245512 13810 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.253283 13708 catalog_manager.cc:5582] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.259248 13880 consensus_queue.cc:237] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.279351 13958 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250623 01:58:25.280169 13853 consensus_peers.cc:487] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adb1e25eef1743848cf218e294bc9226. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:25.282506 14359 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:25.285432 13806 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:25.286892 14359 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:25.290498 14553 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.292078 13958 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.308804 13710 catalog_manager.cc:5582] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:25.316475 13810 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:25.466710 14572 ts_tablet_manager.cc:927] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:25.470131 14572 tablet_copy_client.cc:323] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.2:43055
I20250623 01:58:25.483628 13898 tablet_copy_service.cc:140] P ee041a8325864e788a383b8356afc652: Received BeginTabletCopySession request for tablet 9cb699d5420143efa03140964d5aa739 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:59310)
I20250623 01:58:25.484290 13898 tablet_copy_service.cc:161] P ee041a8325864e788a383b8356afc652: Beginning new tablet copy session on tablet 9cb699d5420143efa03140964d5aa739 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:59310: session id = 7d22741cfb9b408684cb82c58241b2f5-9cb699d5420143efa03140964d5aa739
I20250623 01:58:25.491995 13898 tablet_copy_source_session.cc:215] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:25.495317 14572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb699d5420143efa03140964d5aa739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:25.510599 14572 tablet_copy_client.cc:806] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:25.511260 14572 tablet_copy_client.cc:670] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:25.515045 14572 tablet_copy_client.cc:538] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:25.524607 14572 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:25.547899 14572 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:25.548777 14572 tablet_bootstrap.cc:492] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:25.549446 14572 ts_tablet_manager.cc:1397] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.006s
I20250623 01:58:25.551985 14572 raft_consensus.cc:357] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.552760 14572 raft_consensus.cc:738] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:25.553419 14572 consensus_queue.cc:260] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.557343 14572 ts_tablet_manager.cc:1428] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250623 01:58:25.559531 13898 tablet_copy_service.cc:342] P ee041a8325864e788a383b8356afc652: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-9cb699d5420143efa03140964d5aa739 received from {username='slave'} at 127.0.0.1:59310
I20250623 01:58:25.559971 13898 tablet_copy_service.cc:434] P ee041a8325864e788a383b8356afc652: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-9cb699d5420143efa03140964d5aa739 on tablet 9cb699d5420143efa03140964d5aa739 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:25.626434 14572 ts_tablet_manager.cc:927] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:25.628016 14572 tablet_copy_client.cc:323] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.2:43055
I20250623 01:58:25.629370 13898 tablet_copy_service.cc:140] P ee041a8325864e788a383b8356afc652: Received BeginTabletCopySession request for tablet 433978dce7d94bcb8968bc09583357c4 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:59310)
I20250623 01:58:25.629856 13898 tablet_copy_service.cc:161] P ee041a8325864e788a383b8356afc652: Beginning new tablet copy session on tablet 433978dce7d94bcb8968bc09583357c4 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:59310: session id = 7d22741cfb9b408684cb82c58241b2f5-433978dce7d94bcb8968bc09583357c4
I20250623 01:58:25.635458 13898 tablet_copy_source_session.cc:215] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:25.637818 14572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 433978dce7d94bcb8968bc09583357c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:25.649849 14572 tablet_copy_client.cc:806] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:25.650406 14572 tablet_copy_client.cc:670] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:25.654645 14572 tablet_copy_client.cc:538] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:25.662489 14572 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:25.685195 14572 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:25.686004 14572 tablet_bootstrap.cc:492] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:25.686539 14572 ts_tablet_manager.cc:1397] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.000s
I20250623 01:58:25.688668 14572 raft_consensus.cc:357] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.689293 14572 raft_consensus.cc:738] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:25.689885 14572 consensus_queue.cc:260] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.694092 14572 ts_tablet_manager.cc:1428] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20250623 01:58:25.696976 14579 ts_tablet_manager.cc:927] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:25.698454 13898 tablet_copy_service.cc:342] P ee041a8325864e788a383b8356afc652: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-433978dce7d94bcb8968bc09583357c4 received from {username='slave'} at 127.0.0.1:59310
I20250623 01:58:25.698907 13898 tablet_copy_service.cc:434] P ee041a8325864e788a383b8356afc652: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-433978dce7d94bcb8968bc09583357c4 on tablet 433978dce7d94bcb8968bc09583357c4 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:25.704947 14579 tablet_copy_client.cc:323] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.1:34781
I20250623 01:58:25.716349 13823 tablet_copy_service.cc:140] P 85a58a834b6b4b0dbe197b3dad87e382: Received BeginTabletCopySession request for tablet 2540092c63cb4d0cbe07ae4018f99f46 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:41526)
I20250623 01:58:25.716951 13823 tablet_copy_service.cc:161] P 85a58a834b6b4b0dbe197b3dad87e382: Beginning new tablet copy session on tablet 2540092c63cb4d0cbe07ae4018f99f46 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:41526: session id = 7d22741cfb9b408684cb82c58241b2f5-2540092c63cb4d0cbe07ae4018f99f46
I20250623 01:58:25.723959 13823 tablet_copy_source_session.cc:215] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:25.726905 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2540092c63cb4d0cbe07ae4018f99f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:25.728796 14572 ts_tablet_manager.cc:927] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:25.730463 14572 tablet_copy_client.cc:323] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.2:43055
I20250623 01:58:25.731920 13898 tablet_copy_service.cc:140] P ee041a8325864e788a383b8356afc652: Received BeginTabletCopySession request for tablet adb1e25eef1743848cf218e294bc9226 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:59310)
I20250623 01:58:25.732455 13898 tablet_copy_service.cc:161] P ee041a8325864e788a383b8356afc652: Beginning new tablet copy session on tablet adb1e25eef1743848cf218e294bc9226 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:59310: session id = 7d22741cfb9b408684cb82c58241b2f5-adb1e25eef1743848cf218e294bc9226
I20250623 01:58:25.739466 13898 tablet_copy_source_session.cc:215] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:25.742218 14579 tablet_copy_client.cc:806] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:25.742576 14572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb1e25eef1743848cf218e294bc9226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:25.742808 14579 tablet_copy_client.cc:670] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:25.748654 14579 tablet_copy_client.cc:538] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:25.757192 14579 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:25.762794 14572 tablet_copy_client.cc:806] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:25.763319 14572 tablet_copy_client.cc:670] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:25.767580 14572 tablet_copy_client.cc:538] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:25.774153 14572 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:25.779973 14579 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:25.780820 14579 tablet_bootstrap.cc:492] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:25.781452 14579 ts_tablet_manager.cc:1397] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.006s
I20250623 01:58:25.786541 14579 raft_consensus.cc:357] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.787184 14579 raft_consensus.cc:738] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:25.787672 14579 consensus_queue.cc:260] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.789985 14579 ts_tablet_manager.cc:1428] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20250623 01:58:25.800172 13823 tablet_copy_service.cc:342] P 85a58a834b6b4b0dbe197b3dad87e382: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-2540092c63cb4d0cbe07ae4018f99f46 received from {username='slave'} at 127.0.0.1:41526
I20250623 01:58:25.800628 13823 tablet_copy_service.cc:434] P 85a58a834b6b4b0dbe197b3dad87e382: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-2540092c63cb4d0cbe07ae4018f99f46 on tablet 2540092c63cb4d0cbe07ae4018f99f46 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:25.801817 14572 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:25.802685 14572 tablet_bootstrap.cc:492] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:25.803252 14572 ts_tablet_manager.cc:1397] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.012s
I20250623 01:58:25.805517 14572 raft_consensus.cc:357] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.806219 14572 raft_consensus.cc:738] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:25.806728 14572 consensus_queue.cc:260] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:25.808949 14572 ts_tablet_manager.cc:1428] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250623 01:58:25.811280 13898 tablet_copy_service.cc:342] P ee041a8325864e788a383b8356afc652: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-adb1e25eef1743848cf218e294bc9226 received from {username='slave'} at 127.0.0.1:59310
I20250623 01:58:25.811674 13898 tablet_copy_service.cc:434] P ee041a8325864e788a383b8356afc652: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-adb1e25eef1743848cf218e294bc9226 on tablet adb1e25eef1743848cf218e294bc9226 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:25.906436 14515 raft_consensus.cc:1215] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:26.100031 14515 raft_consensus.cc:1215] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:26.249269 14515 raft_consensus.cc:1215] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:26.297995 14515 raft_consensus.cc:1215] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:26.521173 14359 raft_consensus.cc:1062] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:26.523238 14359 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:26.528360 13960 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.528362 14515 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.529554 13806 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.530571 14359 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:26.531360 14553 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:26.532016 14547 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:26.550761 14553 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.552578 14515 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.553678 13806 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.564431 13960 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.569996 13710 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:26.590415 14359 raft_consensus.cc:1062] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:26.605945 14359 consensus_queue.cc:237] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:26.618868 13882 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:26.628330 14515 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:26.641047 13960 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.642086 14515 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.640931 14596 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:26.642617 14594 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:26.647706 14359 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:26.651979 14594 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.653514 14515 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:58:26.656674 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 53313.1us
I20250623 01:58:26.661384 13807 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.662590 13810 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:26.665010 14547 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:26.669396 14547 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:26.716921 13710 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:58:26.718972 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 45667.2us
W20250623 01:58:26.719205 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 46134.5us
I20250623 01:58:26.721078 13694 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9cb699d5420143efa03140964d5aa739 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:58:26.736325 13810 raft_consensus.cc:2953] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.789958 14594 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.811326 14600 raft_consensus.cc:1062] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:26.813882 14600 consensus_queue.cc:237] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:26.815002 14514 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.828903 14553 raft_consensus.cc:1062] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:26.831220 13806 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.831645 13711 catalog_manager.cc:5582] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.832597 14553 consensus_queue.cc:237] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:26.848307 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 93436.8us
I20250623 01:58:26.855690 14514 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.858088 14515 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.863860 13956 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.860867 13810 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.870185 14553 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:26.871901 14359 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:26.875628 13882 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.875964 13962 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:26.880227 13960 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.880731 14585 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:26.882368 14606 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:26.898608 14602 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:26.936267 14606 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.938042 13955 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.945871 14602 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:26.948442 13882 consensus_queue.cc:237] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:26.949801 13951 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cb699d5420143efa03140964d5aa739 with delete_type TABLET_DATA_TOMBSTONED (TS c243715fe12445019c33bbeafbaea635 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37408
I20250623 01:58:26.966511 14618 tablet_replica.cc:331] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: stopping tablet replica
I20250623 01:58:26.970100 14618 raft_consensus.cc:2241] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:26.970320 14513 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:26.980401 14618 raft_consensus.cc:2270] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:26.978190 14514 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.976820 13806 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.986228 14594 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:26.989603 14585 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.993861 14594 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.007480 13711 catalog_manager.cc:5582] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:26.996315 14513 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.013087 13888 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:26.968832 13960 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:27.022567 13694 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 433978dce7d94bcb8968bc09583357c4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:58:26.996855 14515 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.027990 13710 catalog_manager.cc:5582] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 3 to 4, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.032234 13962 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.045540 14547 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:27.050906 13803 tablet_service.cc:1515] Processing DeleteTablet for tablet 433978dce7d94bcb8968bc09583357c4 with delete_type TABLET_DATA_TOMBSTONED (TS 85a58a834b6b4b0dbe197b3dad87e382 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48562
I20250623 01:58:27.051627 13960 raft_consensus.cc:2953] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.072443 13711 catalog_manager.cc:5582] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.073473 13888 consensus_queue.cc:237] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:27.075004 14618 ts_tablet_manager.cc:1905] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:58:27.090282 13809 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:27.091780 14608 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:27.090842 14515 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:27.095575 14606 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:27.102425 14618 ts_tablet_manager.cc:1918] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250623 01:58:27.103116 14618 log.cc:1199] T 9cb699d5420143efa03140964d5aa739 P c243715fe12445019c33bbeafbaea635: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/wals/9cb699d5420143efa03140964d5aa739
I20250623 01:58:27.104995 14608 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.108079 13697 catalog_manager.cc:4928] TS c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): tablet 9cb699d5420143efa03140964d5aa739 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:58:27.110893 14515 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.117045 13810 consensus_queue.cc:237] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:27.121376 13809 raft_consensus.cc:2953] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.124805 13710 catalog_manager.cc:5582] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.133507 14620 tablet_replica.cc:331] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:58:27.138504 13951 tablet_service.cc:1515] Processing DeleteTablet for tablet adb1e25eef1743848cf218e294bc9226 with delete_type TABLET_DATA_TOMBSTONED (TS c243715fe12445019c33bbeafbaea635 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37408
I20250623 01:58:27.140414 14618 tablet_replica.cc:331] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: stopping tablet replica
I20250623 01:58:27.140304 14515 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:27.141703 13960 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:27.141942 14620 raft_consensus.cc:2241] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:27.141831 14357 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:27.142597 14620 raft_consensus.cc:2270] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:27.143965 14618 raft_consensus.cc:2241] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:27.144656 14618 raft_consensus.cc:2270] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:27.148381 14602 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:27.149602 14618 ts_tablet_manager.cc:1905] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:58:27.150817 13694 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet adb1e25eef1743848cf218e294bc9226 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:58:27.153725 14514 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.159584 14620 ts_tablet_manager.cc:1905] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:58:27.166973 13709 catalog_manager.cc:5582] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.163949 13960 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.176180 14618 ts_tablet_manager.cc:1918] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250623 01:58:27.176736 14618 log.cc:1199] T adb1e25eef1743848cf218e294bc9226 P c243715fe12445019c33bbeafbaea635: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/wals/adb1e25eef1743848cf218e294bc9226
I20250623 01:58:27.178614 14620 ts_tablet_manager.cc:1918] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250623 01:58:27.178645 13697 catalog_manager.cc:4928] TS c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): tablet adb1e25eef1743848cf218e294bc9226 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:58:27.179332 14620 log.cc:1199] T 433978dce7d94bcb8968bc09583357c4 P 85a58a834b6b4b0dbe197b3dad87e382: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root/wals/433978dce7d94bcb8968bc09583357c4
I20250623 01:58:27.151885 14357 raft_consensus.cc:2953] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:27.181725 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet 2540092c63cb4d0cbe07ae4018f99f46 with delete_type TABLET_DATA_TOMBSTONED (TS ee041a8325864e788a383b8356afc652 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58120
I20250623 01:58:27.185180 14621 tablet_replica.cc:331] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: stopping tablet replica
I20250623 01:58:27.186394 14621 raft_consensus.cc:2241] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:27.200313 14621 raft_consensus.cc:2270] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:27.193428 13695 catalog_manager.cc:4928] TS 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): tablet 433978dce7d94bcb8968bc09583357c4 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:58:27.201150 13695 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 2540092c63cb4d0cbe07ae4018f99f46 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:58:27.204468 14621 ts_tablet_manager.cc:1905] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:58:27.218099 14621 ts_tablet_manager.cc:1918] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250623 01:58:27.218423 14621 log.cc:1199] T 2540092c63cb4d0cbe07ae4018f99f46 P ee041a8325864e788a383b8356afc652: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/wals/2540092c63cb4d0cbe07ae4018f99f46
I20250623 01:58:27.219831 13694 catalog_manager.cc:4928] TS ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): tablet 2540092c63cb4d0cbe07ae4018f99f46 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:58:28.273277 13888 consensus_queue.cc:237] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:28.279700 13960 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:28.280413 13806 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250623 01:58:28.280484 13853 consensus_peers.cc:487] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6eba0cd4da75482ab2fcaad6bb1c17a9. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:28.281536 14608 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:28.282575 14596 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:28.290402 14623 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.292018 13960 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.292001 13806 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.311362 13708 catalog_manager.cc:5582] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:28.331391 13960 consensus_queue.cc:237] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:28.341961 13888 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:28.342584 13806 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:28.343999 14627 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:28.345176 14609 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:28.363858 14609 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.365749 13888 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
W20250623 01:58:28.369374 13927 consensus_peers.cc:487] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 829478b3e1dc4a95a122fcd8be3d2c77. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:28.377789 13810 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.378530 13806 consensus_queue.cc:237] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:28.385902 13709 catalog_manager.cc:5582] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:28.406435 13778 consensus_peers.cc:487] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2a94c049f124a68ad6678cff7f838ac. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:28.409375 13888 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:28.410764 14602 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:28.412516 13955 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:28.413956 14602 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:28.429486 14602 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.430945 13882 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.434342 13955 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.447292 13709 catalog_manager.cc:5582] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:28.449044 13806 consensus_queue.cc:237] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:28.467612 13962 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250623 01:58:28.468096 13778 consensus_peers.cc:487] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e92cd39500a645e79b368d26acb587ed. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:28.468848 14603 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:28.471323 13882 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:28.473047 14603 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:28.477540 14357 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.479141 13955 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.484220 13882 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:28.492066 13709 catalog_manager.cc:5582] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:28.767477 14646 ts_tablet_manager.cc:927] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:28.769476 14646 tablet_copy_client.cc:323] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.2:43055
I20250623 01:58:28.771114 13898 tablet_copy_service.cc:140] P ee041a8325864e788a383b8356afc652: Received BeginTabletCopySession request for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:59310)
I20250623 01:58:28.771627 13898 tablet_copy_service.cc:161] P ee041a8325864e788a383b8356afc652: Beginning new tablet copy session on tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:59310: session id = 7d22741cfb9b408684cb82c58241b2f5-6eba0cd4da75482ab2fcaad6bb1c17a9
I20250623 01:58:28.778204 13898 tablet_copy_source_session.cc:215] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:28.781062 14646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:28.792203 14648 ts_tablet_manager.cc:927] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:28.795229 14646 tablet_copy_client.cc:806] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:28.795835 14646 tablet_copy_client.cc:670] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:28.801580 14646 tablet_copy_client.cc:538] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:28.804086 14648 tablet_copy_client.cc:323] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.3:33369
I20250623 01:58:28.811620 14646 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:28.823529 13972 tablet_copy_service.cc:140] P c243715fe12445019c33bbeafbaea635: Received BeginTabletCopySession request for tablet 829478b3e1dc4a95a122fcd8be3d2c77 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:34194)
I20250623 01:58:28.824134 13972 tablet_copy_service.cc:161] P c243715fe12445019c33bbeafbaea635: Beginning new tablet copy session on tablet 829478b3e1dc4a95a122fcd8be3d2c77 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:34194: session id = 7d22741cfb9b408684cb82c58241b2f5-829478b3e1dc4a95a122fcd8be3d2c77
I20250623 01:58:28.830093 13972 tablet_copy_source_session.cc:215] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:28.832943 14648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829478b3e1dc4a95a122fcd8be3d2c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:28.835275 14646 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:28.836421 14646 tablet_bootstrap.cc:492] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:28.837028 14646 ts_tablet_manager.cc:1397] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.000s
I20250623 01:58:28.839708 14646 raft_consensus.cc:357] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:28.840529 14646 raft_consensus.cc:738] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:28.841141 14646 consensus_queue.cc:260] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:28.844200 14646 ts_tablet_manager.cc:1428] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250623 01:58:28.845779 13898 tablet_copy_service.cc:342] P ee041a8325864e788a383b8356afc652: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-6eba0cd4da75482ab2fcaad6bb1c17a9 received from {username='slave'} at 127.0.0.1:59310
I20250623 01:58:28.846190 13898 tablet_copy_service.cc:434] P ee041a8325864e788a383b8356afc652: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-6eba0cd4da75482ab2fcaad6bb1c17a9 on tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:28.848794 14648 tablet_copy_client.cc:806] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:28.849246 14648 tablet_copy_client.cc:670] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:28.858681 14648 tablet_copy_client.cc:538] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:28.865676 14648 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:28.884188 14648 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:28.884971 14648 tablet_bootstrap.cc:492] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:28.885512 14648 ts_tablet_manager.cc:1397] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.010s
I20250623 01:58:28.887930 14648 raft_consensus.cc:357] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:28.888545 14648 raft_consensus.cc:738] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:28.888993 14648 consensus_queue.cc:260] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:28.891024 14648 ts_tablet_manager.cc:1428] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20250623 01:58:28.892452 13972 tablet_copy_service.cc:342] P c243715fe12445019c33bbeafbaea635: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-829478b3e1dc4a95a122fcd8be3d2c77 received from {username='slave'} at 127.0.0.1:34194
I20250623 01:58:28.892866 13972 tablet_copy_service.cc:434] P c243715fe12445019c33bbeafbaea635: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-829478b3e1dc4a95a122fcd8be3d2c77 on tablet 829478b3e1dc4a95a122fcd8be3d2c77 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:28.947575 14648 ts_tablet_manager.cc:927] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:28.949347 14648 tablet_copy_client.cc:323] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.1:34781
I20250623 01:58:28.950858 13823 tablet_copy_service.cc:140] P 85a58a834b6b4b0dbe197b3dad87e382: Received BeginTabletCopySession request for tablet d2a94c049f124a68ad6678cff7f838ac from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:41526)
I20250623 01:58:28.951416 13823 tablet_copy_service.cc:161] P 85a58a834b6b4b0dbe197b3dad87e382: Beginning new tablet copy session on tablet d2a94c049f124a68ad6678cff7f838ac from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:41526: session id = 7d22741cfb9b408684cb82c58241b2f5-d2a94c049f124a68ad6678cff7f838ac
I20250623 01:58:28.957937 13823 tablet_copy_source_session.cc:215] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:28.960547 14648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a94c049f124a68ad6678cff7f838ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:28.973992 14648 tablet_copy_client.cc:806] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:28.974517 14648 tablet_copy_client.cc:670] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:28.977560 14648 tablet_copy_client.cc:538] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:29.002424 14648 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:29.023524 14648 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:29.024260 14648 tablet_bootstrap.cc:492] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:29.024816 14648 ts_tablet_manager.cc:1397] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.012s
I20250623 01:58:29.027077 14648 raft_consensus.cc:357] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:29.027758 14648 raft_consensus.cc:738] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:29.028376 14648 consensus_queue.cc:260] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:29.030537 14648 ts_tablet_manager.cc:1428] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20250623 01:58:29.032301 13823 tablet_copy_service.cc:342] P 85a58a834b6b4b0dbe197b3dad87e382: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-d2a94c049f124a68ad6678cff7f838ac received from {username='slave'} at 127.0.0.1:41526
I20250623 01:58:29.032622 13823 tablet_copy_service.cc:434] P 85a58a834b6b4b0dbe197b3dad87e382: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-d2a94c049f124a68ad6678cff7f838ac on tablet d2a94c049f124a68ad6678cff7f838ac with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:29.043224 14648 ts_tablet_manager.cc:927] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:29.045090 14648 tablet_copy_client.cc:323] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.1:34781
I20250623 01:58:29.046407 13823 tablet_copy_service.cc:140] P 85a58a834b6b4b0dbe197b3dad87e382: Received BeginTabletCopySession request for tablet e92cd39500a645e79b368d26acb587ed from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:41526)
I20250623 01:58:29.046859 13823 tablet_copy_service.cc:161] P 85a58a834b6b4b0dbe197b3dad87e382: Beginning new tablet copy session on tablet e92cd39500a645e79b368d26acb587ed from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:41526: session id = 7d22741cfb9b408684cb82c58241b2f5-e92cd39500a645e79b368d26acb587ed
I20250623 01:58:29.052871 13823 tablet_copy_source_session.cc:215] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:29.055411 14648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92cd39500a645e79b368d26acb587ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:29.068719 14648 tablet_copy_client.cc:806] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:29.069226 14648 tablet_copy_client.cc:670] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:29.072678 14648 tablet_copy_client.cc:538] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:29.080124 14648 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:29.094662 14648 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:29.095321 14648 tablet_bootstrap.cc:492] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:29.095870 14648 ts_tablet_manager.cc:1397] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20250623 01:58:29.098088 14648 raft_consensus.cc:357] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:29.098856 14648 raft_consensus.cc:738] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:29.099414 14648 consensus_queue.cc:260] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:29.101384 14648 ts_tablet_manager.cc:1428] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20250623 01:58:29.103495 13823 tablet_copy_service.cc:342] P 85a58a834b6b4b0dbe197b3dad87e382: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-e92cd39500a645e79b368d26acb587ed received from {username='slave'} at 127.0.0.1:41526
I20250623 01:58:29.103891 13823 tablet_copy_service.cc:434] P 85a58a834b6b4b0dbe197b3dad87e382: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-e92cd39500a645e79b368d26acb587ed on tablet e92cd39500a645e79b368d26acb587ed with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:29.150108 14514 raft_consensus.cc:1215] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:29.382580 14514 raft_consensus.cc:1215] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:29.396206 14514 raft_consensus.cc:1215] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:29.465538 14514 raft_consensus.cc:1215] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:29.703712 14623 raft_consensus.cc:1062] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:29.705807 14623 consensus_queue.cc:237] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:29.712013 13955 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:29.712013 14514 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:29.713621 14596 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:29.713804 13810 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:29.714458 14608 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:29.716457 14608 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:29.728868 14596 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:29.731385 13809 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:29.743036 13955 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:29.749743 13708 catalog_manager.cc:5582] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:29.758302 14515 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:29.832655 13882 consensus_queue.cc:237] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:29.837870 13958 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:29.838220 14515 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:29.838882 14665 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:29.839673 14608 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:29.847712 14515 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:29.849260 14608 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:29.855998 13694 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:58:29.855414 13958 raft_consensus.cc:2953] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:29.864652 13710 catalog_manager.cc:5582] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 3 to 4, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New cstate: current_term: 1 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:29.877206 13803 tablet_service.cc:1515] Processing DeleteTablet for tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 with delete_type TABLET_DATA_TOMBSTONED (TS 85a58a834b6b4b0dbe197b3dad87e382 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48562
I20250623 01:58:29.880323 14669 tablet_replica.cc:331] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:58:29.881304 14669 raft_consensus.cc:2241] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:29.882027 14669 raft_consensus.cc:2270] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:29.885020 14669 ts_tablet_manager.cc:1905] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:58:29.897935 14669 ts_tablet_manager.cc:1918] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250623 01:58:29.898240 14669 log.cc:1199] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 85a58a834b6b4b0dbe197b3dad87e382: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root/wals/6eba0cd4da75482ab2fcaad6bb1c17a9
I20250623 01:58:29.902045 13695 catalog_manager.cc:4928] TS 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): tablet 6eba0cd4da75482ab2fcaad6bb1c17a9 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:58:29.962821 14639 raft_consensus.cc:1062] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:29.965157 14639 consensus_queue.cc:237] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:29.973188 14515 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:29.974247 13888 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:29.974452 14609 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:29.975574 13809 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:29.975606 14639 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:29.976879 14609 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:29.981603 14658 raft_consensus.cc:1062] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:29.984874 14658 consensus_queue.cc:237] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:30.000905 14603 raft_consensus.cc:1062] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:30.003197 14603 consensus_queue.cc:237] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:30.011353 14676 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.018910 14515 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.026818 13809 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.029281 14514 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:30.029822 13888 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:30.030936 14673 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:30.036585 13711 catalog_manager.cc:5582] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.034425 14673 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:30.043529 13955 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:30.049192 14675 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:30.051144 13882 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.058604 14514 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:30.061483 14673 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:30.067492 13955 consensus_queue.cc:237] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:30.073850 13958 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:30.076004 14682 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:30.081655 14515 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:30.084239 13883 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:30.084751 14676 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:30.089557 14658 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:30.093297 13810 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:30.098934 14609 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.101444 14515 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.103175 13883 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.096758 14673 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.123005 13710 catalog_manager.cc:5582] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.142472 13709 catalog_manager.cc:5582] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.146050 13697 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 829478b3e1dc4a95a122fcd8be3d2c77 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:58:30.153867 14602 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.175805 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet 829478b3e1dc4a95a122fcd8be3d2c77 with delete_type TABLET_DATA_TOMBSTONED (TS ee041a8325864e788a383b8356afc652 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58120
I20250623 01:58:30.174175 14515 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.178632 14690 tablet_replica.cc:331] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: stopping tablet replica
I20250623 01:58:30.181164 14690 raft_consensus.cc:2241] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:30.181885 14690 raft_consensus.cc:2270] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:30.180667 13711 catalog_manager.cc:5582] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:30.188891 14609 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:30.194224 14515 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:58:30.212834 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 62061.7us
I20250623 01:58:30.212399 13882 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.212399 14690 ts_tablet_manager.cc:1905] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:58:30.218189 13958 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.234040 13807 raft_consensus.cc:2953] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.243297 13955 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.245854 14690 ts_tablet_manager.cc:1918] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250623 01:58:30.246250 14690 log.cc:1199] T 829478b3e1dc4a95a122fcd8be3d2c77 P ee041a8325864e788a383b8356afc652: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/wals/829478b3e1dc4a95a122fcd8be3d2c77
I20250623 01:58:30.248111 13810 consensus_queue.cc:237] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:30.248512 13694 catalog_manager.cc:4928] TS ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): tablet 829478b3e1dc4a95a122fcd8be3d2c77 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:58:30.254074 13882 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:30.254622 13807 consensus_queue.cc:237] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:30.255285 14515 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:30.257418 14675 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:30.258533 14688 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:30.260587 14515 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:30.261978 14658 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:30.265288 14603 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.266698 13882 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.273754 14658 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.275529 14514 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.280741 14513 raft_consensus.cc:2953] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.286051 13695 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d2a94c049f124a68ad6678cff7f838ac with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:58:30.286782 13695 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e92cd39500a645e79b368d26acb587ed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:58:30.289495 13958 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250623 01:58:30.291985 14602 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:30.296499 13958 raft_consensus.cc:2953] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.301851 13709 catalog_manager.cc:5582] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 3 to 4, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.306692 13710 catalog_manager.cc:5582] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 3 to 4, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:30.312310 13951 tablet_service.cc:1515] Processing DeleteTablet for tablet d2a94c049f124a68ad6678cff7f838ac with delete_type TABLET_DATA_TOMBSTONED (TS c243715fe12445019c33bbeafbaea635 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37408
I20250623 01:58:30.326251 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet e92cd39500a645e79b368d26acb587ed with delete_type TABLET_DATA_TOMBSTONED (TS ee041a8325864e788a383b8356afc652 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58120
I20250623 01:58:30.330099 14690 tablet_replica.cc:331] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: stopping tablet replica
I20250623 01:58:30.331152 14690 raft_consensus.cc:2241] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:30.331516 14691 tablet_replica.cc:331] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: stopping tablet replica
I20250623 01:58:30.331817 14690 raft_consensus.cc:2270] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:30.336606 14690 ts_tablet_manager.cc:1905] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:58:30.346179 14691 raft_consensus.cc:2241] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 01:58:30.346987 14691 raft_consensus.cc:2270] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:30.350680 14691 ts_tablet_manager.cc:1905] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:58:30.363035 14690 ts_tablet_manager.cc:1918] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250623 01:58:30.363521 14690 log.cc:1199] T e92cd39500a645e79b368d26acb587ed P ee041a8325864e788a383b8356afc652: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/wals/e92cd39500a645e79b368d26acb587ed
I20250623 01:58:30.364768 13694 catalog_manager.cc:4928] TS ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): tablet e92cd39500a645e79b368d26acb587ed (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:58:30.369853 14691 ts_tablet_manager.cc:1918] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250623 01:58:30.370208 14691 log.cc:1199] T d2a94c049f124a68ad6678cff7f838ac P c243715fe12445019c33bbeafbaea635: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/wals/d2a94c049f124a68ad6678cff7f838ac
I20250623 01:58:30.371443 13697 catalog_manager.cc:4928] TS c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): tablet d2a94c049f124a68ad6678cff7f838ac (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:58:31.431354 13960 consensus_queue.cc:237] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:31.438982 13882 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250623 01:58:31.439550 13927 consensus_peers.cc:487] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b39b05807ed544d388835cc364cb7274. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:31.440512 14692 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:31.441468 13807 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:31.442515 14692 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:31.447777 14609 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.449285 13882 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.450704 13807 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.466905 13809 consensus_queue.cc:237] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
W20250623 01:58:31.500710 13778 consensus_peers.cc:487] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 272c03e6252d489cbf2fdceb36b05dc1. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:31.504256 13955 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:31.504882 13880 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:31.492340 13710 catalog_manager.cc:5582] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:31.510577 14698 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:31.511736 14602 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:31.520416 14698 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.522599 13955 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.524830 13880 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.536726 13955 consensus_queue.cc:237] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:31.545605 13807 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:31.546898 14609 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:31.547581 13880 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:31.549357 14609 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:31.550988 13709 catalog_manager.cc:5582] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:31.554140 13927 consensus_peers.cc:487] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de00fb2e24074586bfe02d33fb92e4e6. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:31.568264 14692 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.570837 13807 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.571863 13882 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.588378 13710 catalog_manager.cc:5582] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.588598 13955 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
W20250623 01:58:31.604717 13927 consensus_peers.cc:487] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f12346da5a14097bbfa150cb8d1d907. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:31.610996 13806 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:31.613871 13882 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:58:31.615707 14692 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:31.616731 14609 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:31.633236 14711 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.635090 13807 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.640411 13880 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.662313 13709 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index -1 to 2, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:58:31.964013 14714 ts_tablet_manager.cc:927] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:31.966105 14714 tablet_copy_client.cc:323] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.3:33369
I20250623 01:58:31.968014 13972 tablet_copy_service.cc:140] P c243715fe12445019c33bbeafbaea635: Received BeginTabletCopySession request for tablet de00fb2e24074586bfe02d33fb92e4e6 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:34194)
I20250623 01:58:31.968561 13972 tablet_copy_service.cc:161] P c243715fe12445019c33bbeafbaea635: Beginning new tablet copy session on tablet de00fb2e24074586bfe02d33fb92e4e6 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:34194: session id = 7d22741cfb9b408684cb82c58241b2f5-de00fb2e24074586bfe02d33fb92e4e6
I20250623 01:58:31.975253 13972 tablet_copy_source_session.cc:215] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:31.978433 14714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de00fb2e24074586bfe02d33fb92e4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:31.995870 14714 tablet_copy_client.cc:806] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:31.996465 14714 tablet_copy_client.cc:670] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:32.000460 14714 tablet_copy_client.cc:538] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:32.008414 14714 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:32.015652 14716 ts_tablet_manager.cc:927] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:32.017725 14716 tablet_copy_client.cc:323] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.1:34781
I20250623 01:58:32.019572 13823 tablet_copy_service.cc:140] P 85a58a834b6b4b0dbe197b3dad87e382: Received BeginTabletCopySession request for tablet 272c03e6252d489cbf2fdceb36b05dc1 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:41526)
I20250623 01:58:32.020095 13823 tablet_copy_service.cc:161] P 85a58a834b6b4b0dbe197b3dad87e382: Beginning new tablet copy session on tablet 272c03e6252d489cbf2fdceb36b05dc1 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:41526: session id = 7d22741cfb9b408684cb82c58241b2f5-272c03e6252d489cbf2fdceb36b05dc1
I20250623 01:58:32.031783 13823 tablet_copy_source_session.cc:215] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:32.033979 14719 ts_tablet_manager.cc:927] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:32.035789 14716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272c03e6252d489cbf2fdceb36b05dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:32.035895 14719 tablet_copy_client.cc:323] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.3:33369
I20250623 01:58:32.038339 14714 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:32.039428 13972 tablet_copy_service.cc:140] P c243715fe12445019c33bbeafbaea635: Received BeginTabletCopySession request for tablet b39b05807ed544d388835cc364cb7274 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:34194)
I20250623 01:58:32.039952 13972 tablet_copy_service.cc:161] P c243715fe12445019c33bbeafbaea635: Beginning new tablet copy session on tablet b39b05807ed544d388835cc364cb7274 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:34194: session id = 7d22741cfb9b408684cb82c58241b2f5-b39b05807ed544d388835cc364cb7274
I20250623 01:58:32.045605 13972 tablet_copy_source_session.cc:215] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:32.039187 14714 tablet_bootstrap.cc:492] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:32.047610 14714 ts_tablet_manager.cc:1397] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.039s user 0.022s sys 0.001s
I20250623 01:58:32.050585 14714 raft_consensus.cc:357] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:32.051333 14714 raft_consensus.cc:738] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:32.052855 14714 consensus_queue.cc:260] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:32.060194 14716 tablet_copy_client.cc:806] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:32.060792 14719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b39b05807ed544d388835cc364cb7274. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:32.060842 14716 tablet_copy_client.cc:670] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:32.062640 14714 ts_tablet_manager.cc:1428] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20250623 01:58:32.064954 13972 tablet_copy_service.cc:342] P c243715fe12445019c33bbeafbaea635: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-de00fb2e24074586bfe02d33fb92e4e6 received from {username='slave'} at 127.0.0.1:34194
I20250623 01:58:32.065429 13972 tablet_copy_service.cc:434] P c243715fe12445019c33bbeafbaea635: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-de00fb2e24074586bfe02d33fb92e4e6 on tablet de00fb2e24074586bfe02d33fb92e4e6 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:32.067509 14716 tablet_copy_client.cc:538] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:32.081128 14716 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:32.085875 14719 tablet_copy_client.cc:806] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:32.086453 14719 tablet_copy_client.cc:670] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:32.090250 14719 tablet_copy_client.cc:538] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:32.098026 14719 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:32.102535 14716 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:32.103260 14716 tablet_bootstrap.cc:492] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:32.103895 14716 ts_tablet_manager.cc:1397] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.005s
I20250623 01:58:32.106767 14716 raft_consensus.cc:357] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:32.107481 14716 raft_consensus.cc:738] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:32.108064 14716 consensus_queue.cc:260] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:32.110802 14716 ts_tablet_manager.cc:1428] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250623 01:58:32.112525 13823 tablet_copy_service.cc:342] P 85a58a834b6b4b0dbe197b3dad87e382: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-272c03e6252d489cbf2fdceb36b05dc1 received from {username='slave'} at 127.0.0.1:41526
I20250623 01:58:32.112890 13823 tablet_copy_service.cc:434] P 85a58a834b6b4b0dbe197b3dad87e382: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-272c03e6252d489cbf2fdceb36b05dc1 on tablet 272c03e6252d489cbf2fdceb36b05dc1 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:32.120859 14719 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:32.121570 14719 tablet_bootstrap.cc:492] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:32.122061 14719 ts_tablet_manager.cc:1397] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.004s
I20250623 01:58:32.124559 14719 raft_consensus.cc:357] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:32.125187 14719 raft_consensus.cc:738] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:32.125766 14719 consensus_queue.cc:260] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:32.127820 14719 ts_tablet_manager.cc:1428] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250623 01:58:32.129266 13972 tablet_copy_service.cc:342] P c243715fe12445019c33bbeafbaea635: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-b39b05807ed544d388835cc364cb7274 received from {username='slave'} at 127.0.0.1:34194
I20250623 01:58:32.129690 13972 tablet_copy_service.cc:434] P c243715fe12445019c33bbeafbaea635: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-b39b05807ed544d388835cc364cb7274 on tablet b39b05807ed544d388835cc364cb7274 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:32.180564 14719 ts_tablet_manager.cc:927] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:32.183107 14719 tablet_copy_client.cc:323] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.3:33369
I20250623 01:58:32.184371 13972 tablet_copy_service.cc:140] P c243715fe12445019c33bbeafbaea635: Received BeginTabletCopySession request for tablet 0f12346da5a14097bbfa150cb8d1d907 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:34194)
I20250623 01:58:32.184825 13972 tablet_copy_service.cc:161] P c243715fe12445019c33bbeafbaea635: Beginning new tablet copy session on tablet 0f12346da5a14097bbfa150cb8d1d907 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:34194: session id = 7d22741cfb9b408684cb82c58241b2f5-0f12346da5a14097bbfa150cb8d1d907
I20250623 01:58:32.190130 13972 tablet_copy_source_session.cc:215] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:32.193426 14719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f12346da5a14097bbfa150cb8d1d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:32.207331 14719 tablet_copy_client.cc:806] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:32.207867 14719 tablet_copy_client.cc:670] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:32.221405 14719 tablet_copy_client.cc:538] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:32.230948 14719 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:58:32.254374 14719 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:32.255242 14719 tablet_bootstrap.cc:492] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:58:32.255852 14719 ts_tablet_manager.cc:1397] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.008s
I20250623 01:58:32.258339 14719 raft_consensus.cc:357] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:32.258909 14719 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:58:32.259518 14719 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:32.261924 14719 ts_tablet_manager.cc:1428] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250623 01:58:32.263386 13972 tablet_copy_service.cc:342] P c243715fe12445019c33bbeafbaea635: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-0f12346da5a14097bbfa150cb8d1d907 received from {username='slave'} at 127.0.0.1:34194
I20250623 01:58:32.263782 13972 tablet_copy_service.cc:434] P c243715fe12445019c33bbeafbaea635: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-0f12346da5a14097bbfa150cb8d1d907 on tablet 0f12346da5a14097bbfa150cb8d1d907 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:32.460984 14513 raft_consensus.cc:1215] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:32.513979 14513 raft_consensus.cc:1215] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:33.379343 14732 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:33.385690 14732 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.387522 14732 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:33.495414 12641 debug-util.cc:398] Leaking SignalData structure 0x7b080000e9c0 after lost signal to thread 13776
W20250623 01:58:33.496104 12641 kernel_stack_watchdog.cc:198] Thread 13776 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 115ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20250623 01:58:33.530773 14729 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.531595 14729 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:33.534245 14729 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:33.547161 14732 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:33.547932 14732 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.549710 14729 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.550444 14729 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.552809 14729 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.553791 14700 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:33.567239 14700 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:33.570164 14700 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.561650 14732 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:33.592950 13742 debug-util.cc:398] Leaking SignalData structure 0x7b0800036460 after lost signal to thread 12639
I20250623 01:58:33.598541 14733 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.599287 14733 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.606859 14729 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:33.607620 14729 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:33.625217 14728 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.626051 14728 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:33.628587 14728 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:33.647284 14711 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.648156 14711 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.650799 14711 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:33.651506 14711 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:33.652415 14711 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.698087 14700 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.698918 14700 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.701480 14700 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.702080 14700 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.702620 14700 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.718660 14732 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:33.719638 14733 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.720409 14733 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:33.737718 14728 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.738500 14728 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.751135 14728 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.751809 14728 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.752429 14728 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.780085 14711 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:33.780997 14711 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.720975 14733 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:33.847517 14733 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.849253 14733 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.781723 14711 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.062006 14733 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.063540 14711 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.064038 14733 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.064630 14733 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.904961 14729 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.936054 14728 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.950482 14735 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.978754 14700 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:33.985459 14736 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.986826 14737 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.024261 14734 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.030577 14596 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.032884 14739 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.035847 14692 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:58:34.036214 13776 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.656s user 0.020s sys 0.013s
W20250623 01:58:34.080248 13776 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.700s user 0.021s sys 0.013s
W20250623 01:58:34.080686 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 1.70089e+06us
I20250623 01:58:34.055933 14744 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.082487 14692 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.059036 14743 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:33.719458 14732 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.079501 14596 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.082756 14739 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.082583 14744 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.100284 14748 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.100775 14748 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.102408 14748 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.104208 14748 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.080017 14736 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.129161 14596 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.132295 14596 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.136449 14596 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.082376 14735 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.143388 14735 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.144397 14596 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.145882 14596 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.146426 14596 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.146533 14692 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.096170 14733 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.155678 14749 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.156303 14749 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:34.158617 14749 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.156317 14596 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.098743 14737 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.166219 14737 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.098524 14743 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.168808 14743 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.169402 14743 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.149296 14692 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.176133 14751 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.147851 14739 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:34.095793 14734 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.161904 14735 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.183065 14745 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.137658 14748 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.176134 14692 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.170950 14743 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.207067 14743 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.207130 14692 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.201591 14596 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.195492 14711 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.176759 14751 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.191282 14735 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.193599 14745 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.251157 14733 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.302498 14756 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.302426 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:34.300045 14596 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.285912 14734 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.211244 14736 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.320255 14596 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.317212 14747 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.250558 14754 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.205363 14732 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.294127 14752 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.285391 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:34.321846 14747 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.326419 14512 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:34.277805 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:34.299803 14711 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.301635 14737 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.329789 14721 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.251794 14748 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.236297 14744 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.323354 14515 raft_consensus.cc:2466] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:34.330526 14737 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.303110 14756 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.221827 14757 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.332315 14758 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:58:34.327529 13742 debug-util.cc:398] Leaking SignalData structure 0x7b0800036b40 after lost signal to thread 13745
I20250623 01:58:34.330706 14721 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.332899 14752 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.330914 14747 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.340103 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 849) took 1837 ms. Trace:
I20250623 01:58:34.337786 14737 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.322865 14754 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.347563 14511 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:34.354652 13778 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:58:34.341564 13807 rpcz_store.cc:276] 0623 01:58:32.502275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:34.339358 (+1837083us) service_pool.cc:225] Handling call
0623 01:58:34.339660 (+ 302us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:34.339811 (+ 151us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:34.339834 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:34.339874 (+ 40us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:34.339901 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:34.339952 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:34.340046 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:34.370194 14745 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.370069 14746 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.370908 14745 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.371397 14745 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.371039 14746 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.373459 14692 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.375532 14692 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.340947 14757 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.381021 14745 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.386155 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.389446 14692 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.390516 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 53) took 1900 ms. Trace:
I20250623 01:58:34.342149 14758 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.378405 14756 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.394001 14596 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.381583 14745 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.381067 14743 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.394882 14596 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.369588 14744 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:34.390859 13711 rpcz_store.cc:276] 0623 01:58:32.486913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:32.487767 (+ 854us) service_pool.cc:225] Handling call
0623 01:58:34.387644 (+1899877us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20250623 01:58:34.397322 14753 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.397900 14747 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.398440 14748 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.399313 14755 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.400974 14738 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.401360 14665 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.390713 14692 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.393579 14735 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.396939 14756 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:58:34.406783 13742 debug-util.cc:398] Leaking SignalData structure 0x7b080020c800 after lost signal to thread 13837
I20250623 01:58:34.406721 14721 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.408736 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.391343 14734 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.413761 14744 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.407958 14761 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.432240 14751 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.395586 14749 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.433028 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.424018 14738 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.392752 14754 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.436564 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.437425 14596 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.438614 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.434887 14753 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.425853 14732 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.475414 14596 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.474938 13809 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:34.443341 14761 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.424528 14757 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:34.473183 13807 raft_consensus.cc:2466] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
W20250623 01:58:34.439407 13742 debug-util.cc:398] Leaking SignalData structure 0x7b08002eb8a0 after lost signal to thread 13838
I20250623 01:58:34.421716 14743 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.433341 14711 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.413913 14745 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.427042 14747 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.424855 14748 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.443950 14744 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:34.480149 14596 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.443969 14760 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.485867 14753 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.492405 14740 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.492921 14479 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:58:34.434029 14665 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:34.498536 12641 debug-util.cc:398] Leaking SignalData structure 0x7b080000ea20 after lost signal to thread 13776
I20250623 01:58:34.473588 14734 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.483340 14749 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.487571 14754 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.474525 14755 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.479596 14752 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.509874 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.487442 14757 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.510778 14602 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.459354 14746 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.436749 14692 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:34.512451 14738 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.515782 14596 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.441921 14756 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.491820 14711 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.483695 14732 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:34.526844 14596 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.507232 14753 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.533214 14596 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:34.537488 14602 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.531360 14752 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:34.544049 14746 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.545485 14698 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.545614 14728 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.545504 14700 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.546028 14736 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.546453 14749 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.538125 14711 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:34.526782 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 46678.1us
I20250623 01:58:34.547672 14748 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.548411 14665 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.549798 14732 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.531239 14738 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:58:34.543349 13742 debug-util.cc:398] Leaking SignalData structure 0x7b08002f4100 after lost signal to thread 13912
I20250623 01:58:34.555083 14737 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.536887 14754 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.574975 14602 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.590613 14596 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:34.551187 14752 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.508329 14760 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.524626 14740 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.528466 14757 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.557709 14750 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.557701 14759 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.622670 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.583138 14762 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.628012 13955 raft_consensus.cc:2466] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:34.587383 14749 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.591889 14748 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.560910 14700 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.641052 14766 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.623374 14732 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.641574 14761 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.589084 14739 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.640816 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.620534 14752 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:34.573858 14746 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.574394 14736 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.619819 14743 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.651530 14765 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.651983 14700 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.642736 13956 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:34.653576 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.627853 14758 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.575037 14698 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.642729 14749 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.558878 14753 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.652107 14765 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.651985 14739 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.663358 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.656244 14767 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.641062 14764 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.641923 14732 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.645574 14761 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.619493 14728 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.661855 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.662503 14740 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:34.683100 14763 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.688433 14749 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.656699 14711 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:34.620947 14737 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:34.654174 13742 debug-util.cc:398] Leaking SignalData structure 0x7b080029d360 after lost signal to thread 13986
I20250623 01:58:34.623262 14479 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:58:34.645943 14745 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.623835 14738 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.646234 14747 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.682617 14755 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.683089 14739 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.683182 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.644590 14748 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.703909 14736 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.710947 13962 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:34.704339 14698 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.704641 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.677992 14734 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.708684 14769 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.709179 14741 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.709713 14728 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.713066 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.688601 14763 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.721015 13960 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:34.636620 14759 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.714946 14745 raft_consensus.cc:2802] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:34.710194 14756 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.641589 14766 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.698482 14764 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:34.698189 14767 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:34.713137 14692 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.690732 14761 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.717576 14754 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:34.730701 14481 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:58:34.715229 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 96261.2us
I20250623 01:58:34.714828 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:34.731775 13742 debug-util.cc:398] Leaking SignalData structure 0x7b080026bbc0 after lost signal to thread 14540
I20250623 01:58:34.732196 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.725369 14745 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.739269 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.720500 13885 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.744419 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.744807 14774 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.745056 13886 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.745240 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.659775 14750 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:34.657079 14762 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.745823 13954 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:34.748257 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.753832 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.740235 14753 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.746313 14756 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.755057 13961 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:34.736114 14743 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.757651 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.761658 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.759997 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.746486 14757 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.753345 14737 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.743544 14745 raft_consensus.cc:3058] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:34.753835 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.763587 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:34.754410 14732 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.762187 13955 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:34.771653 14766 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.764931 14746 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.766068 14511 raft_consensus.cc:1215] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:34.778285 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.760972 14768 raft_consensus.cc:2802] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:34.786240 14768 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.785861 14748 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.786700 14768 raft_consensus.cc:3058] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
W20250623 01:58:34.764194 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 44128us
I20250623 01:58:34.764997 14711 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.745576 14774 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.777421 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.782946 14733 raft_consensus.cc:1062] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:34.783839 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.766669 14756 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.777243 14776 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.764303 14744 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.737344 14741 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.790941 14515 raft_consensus.cc:1215] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250623 01:58:34.767975 13957 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:34.803829 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.804363 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.804900 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.805524 13882 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:34.806051 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.806282 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.806852 13882 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.807171 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.797130 14711 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.807780 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.808265 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.808946 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.809481 14711 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.774848 13879 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.793004 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.796257 14481 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:34.767720 14743 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.785832 14761 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.775950 14734 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.734207 14769 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.773592 14767 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.807773 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.811344 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.811430 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.812829 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.813251 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.789706 14738 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.810331 14757 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.828483 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.810829 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.811204 14692 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.829447 13810 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.779955 14765 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.830355 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.830641 14756 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.831492 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.832062 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.819535 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.831322 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.832640 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.825676 14751 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:34.833127 13809 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.833418 14751 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.819938 14762 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:34.797912 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.821781 13807 raft_consensus.cc:1238] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:34.824164 13888 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.790491 14760 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.810153 14763 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.801015 14746 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.812072 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.828907 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.810653 14732 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.818262 14750 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.833853 14751 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:34.835997 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.820529 14734 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.835999 14759 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.843813 13812 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.844609 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.845665 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.845688 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.846921 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.848738 14732 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.849112 14774 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.837471 14757 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.852003 14746 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.836426 14692 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.863610 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.867359 14744 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:34.826157 14764 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.879549 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.818056 14776 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.880306 14744 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.882088 14744 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:34.883460 14737 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.884117 14737 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.886696 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.897444 14733 consensus_queue.cc:237] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.903311 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.919675 14759 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:34.920323 14759 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:34.924427 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.924726 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:34.925776 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:34.927404 14757 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.928156 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:34.928267 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 80754us
I20250623 01:58:34.928730 13958 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:34.929030 14481 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:34.930588 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.931015 13958 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:34.928119 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.933568 14771 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:34.933784 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.933869 14759 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.934010 14771 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:34.935360 14771 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:34.937259 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.937814 13812 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:34.940058 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.945183 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.945739 13955 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:34.946024 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.940021 14743 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.944849 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.949709 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.949957 14752 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.950335 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.951022 14513 raft_consensus.cc:1238] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:34.947145 14737 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.952595 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.952253 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.953763 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:34.953789 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.943706 14732 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.953917 14776 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.965907 14771 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:34.967864 14771 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.968456 14769 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.954298 14741 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.975099 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:34.976310 14734 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:34.976437 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:34.977772 14744 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:34.952857 14481 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:58:34.995384 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 51191.1us
I20250623 01:58:34.968474 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:34.996676 14771 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:34.997087 14771 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:34.997515 14771 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:34.999553 14692 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:35.003396 14752 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:35.006660 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.007298 13960 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:35.007707 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 58748.6us
I20250623 01:58:35.006659 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.008967 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.010339 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:35.013053 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.013430 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.013713 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.014274 13955 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.014195 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.013721 13957 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.007540 14751 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.023626 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.016135 14746 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:35.029779 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.030120 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.030543 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.004273 14768 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.061069 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.022583 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.059139 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.031317 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.064693 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:35.032711 14480 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.042s sys 0.042s
W20250623 01:58:35.065651 14480 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.043s sys 0.043s
I20250623 01:58:35.066368 14510 raft_consensus.cc:1238] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:35.067142 14768 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.020573 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.058676 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.068598 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.069213 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.069525 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.069615 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.064626 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.070148 13960 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.069229 13957 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.070256 13955 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.071823 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.072623 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:35.072737 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:35.061674 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.074040 13808 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.073655 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.074877 13960 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.075343 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.075531 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:35.076174 14478 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.031298 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.077709 14751 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.077873 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:35.078269 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.078395 13955 raft_consensus.cc:3058] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.079023 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.079408 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.080111 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.079452 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.081545 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.082603 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.083149 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.083933 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.031638 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.084522 13955 raft_consensus.cc:2466] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:35.084407 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.078328 14751 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.087177 14481 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:35.087922 14768 raft_consensus.cc:2802] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:35.088002 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.088148 14478 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.088431 14768 raft_consensus.cc:695] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:35.089186 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:35.089408 14478 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.089301 14768 consensus_queue.cc:237] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.090056 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.091786 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.092756 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.102123 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.093569 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.093643 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.104737 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.105402 13805 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:35.106312 13778 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:58:35.108273 13778 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:58:35.108120 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.109649 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.109880 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.110430 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.110514 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.102226 13806 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.091369 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.112921 13961 raft_consensus.cc:1238] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:35.114091 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.114925 13958 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.103036 14751 raft_consensus.cc:2747] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.094310 14752 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.117588 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.117842 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.118014 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.109678 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.120594 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.119225 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.119568 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.119725 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.118577 13810 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:35.121850 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.114231 14733 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.112520 14744 raft_consensus.cc:2802] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.093256 14771 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.124473 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.088143 14745 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.116066 14737 consensus_queue.cc:1046] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20250623 01:58:35.171226 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.160815 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.139750 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.140120 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.141618 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.140856 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.148381 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.141376 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.127048 13854 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:35.158952 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.159660 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.179595 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:35.179677 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 53391.2us
I20250623 01:58:35.180569 13854 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:35.178755 14692 raft_consensus.cc:1062] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:35.181655 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.174978 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.183476 14764 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.183876 14764 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.184230 14764 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.184885 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.185653 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:35.185263 14692 consensus_queue.cc:237] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.187176 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.188369 14759 raft_consensus.cc:3053] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.188781 14759 raft_consensus.cc:738] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:35.158918 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.189519 14512 raft_consensus.cc:1238] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:35.190169 14759 consensus_queue.cc:260] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.191529 14759 raft_consensus.cc:3058] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.127223 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:35.133769 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.127401 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.195766 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.196312 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.197081 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.198004 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.123831 14733 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:35.140518 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.170436 14771 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:35.204865 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:35.126447 14744 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:35.202885 14479 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:35.203425 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.134465 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.204077 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.203830 14733 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.213897 13854 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:35.215160 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.206578 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:35.209079 13881 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:35.208010 14744 raft_consensus.cc:3058] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.224848 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.227735 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:35.215371 14771 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.229715 14761 consensus_queue.cc:1046] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:35.231256 14761 raft_consensus.cc:3053] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.231593 14761 raft_consensus.cc:738] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:35.232447 14761 consensus_queue.cc:260] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.233610 14761 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.234378 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.212316 14767 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.236400 14767 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.236780 14767 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.239224 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:35.245905 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:35.246378 14511 raft_consensus.cc:1238] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:35.235242 14750 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.247591 14510 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:35.247880 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.249953 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.251873 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.255656 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.252473 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.255957 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.247639 14750 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.253731 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.254335 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.254549 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.253036 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.258234 14750 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.258158 14734 consensus_queue.cc:1046] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:58:35.262476 13852 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:35.263270 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.253141 13709 catalog_manager.cc:5582] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 2, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 2 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:35.265372 13776 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:35.273689 14746 raft_consensus.cc:3053] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.274080 14746 raft_consensus.cc:738] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:35.274864 14746 consensus_queue.cc:260] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.275951 14746 raft_consensus.cc:3058] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.267467 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.277845 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.278527 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.279532 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.285619 13958 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:35.288072 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.289158 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.291608 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.294370 14736 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.294554 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.294818 14736 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:35.294862 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:35.295325 14736 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.267580 14748 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.294499 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.298717 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.298748 13854 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.301209 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:35.303112 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:35.297896 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.305931 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.304844 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.265170 14764 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.313439 13926 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:35.302237 14745 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.324488 13926 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:35.325506 13926 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:35.326992 13926 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:35.314555 13808 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:35.330559 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:35.330583 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:35.321044 14743 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.321417 14760 consensus_queue.cc:1046] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20250623 01:58:35.341004 14786 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.317416 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:35.361279 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 89783.9us
I20250623 01:58:35.332093 14733 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.366863 14760 raft_consensus.cc:3053] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.367247 14760 raft_consensus.cc:738] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:35.368180 14733 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:35.367978 14760 consensus_queue.cc:260] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.332557 14732 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.339767 14774 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.332677 14757 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.362695 13926 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:35.364749 14510 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250623 01:58:35.365262 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 66584.2us
I20250623 01:58:35.374086 13926 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.358366 14743 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:35.359524 14786 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.369253 14760 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
W20250623 01:58:35.387996 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 76279.8us
I20250623 01:58:35.371570 14732 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:35.374765 14769 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.371733 14774 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:35.384038 14743 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.381806 14734 raft_consensus.cc:2747] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.373124 14757 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:35.386242 14764 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.387177 14786 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.370568 14728 consensus_queue.cc:1046] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20250623 01:58:35.382221 14733 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20250623 01:58:35.392617 14732 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.406131 13854 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.399161 14757 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.407636 14772 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.396163 14774 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.395645 14741 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20250623 01:58:35.395068 14769 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:35.404397 14750 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.421900 14750 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.424960 14786 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.425833 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:35.427220 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.417225 14736 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.429579 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.410778 14772 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.413996 14769 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.445019 14789 raft_consensus.cc:3053] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.445050 14772 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.453400 14510 raft_consensus.cc:2391] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:35.454762 13776 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
W20250623 01:58:35.462936 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 61211.9us
I20250623 01:58:35.445487 14789 raft_consensus.cc:738] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:35.458395 14728 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.466864 14789 consensus_queue.cc:260] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:35.467834 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 81785.6us
W20250623 01:58:35.468097 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 62044.4us
I20250623 01:58:35.468910 13854 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.463937 14770 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20250623 01:58:35.473802 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:35.474745 14692 raft_consensus.cc:1062] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:35.476205 14776 raft_consensus.cc:1062] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:58:35.477074 14692 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.478392 14776 consensus_queue.cc:237] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.490217 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.496582 14786 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.506606 13885 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:35.506906 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:35.487358 14795 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.512305 14795 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.512718 14795 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.514407 13852 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:35.516599 14793 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.513032 14736 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.503580 14760 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.486398 14744 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:35.516992 14698 consensus_queue.cc:1046] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:35.518784 13852 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:35.496606 14767 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.513756 14794 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.519897 14760 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.530902 14760 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.531071 14757 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.531683 14789 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.533432 13810 raft_consensus.cc:2391] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:35.535666 13854 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.517282 14793 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:35.539090 13888 raft_consensus.cc:1238] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:35.538434 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:35.542193 13854 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:35.535101 14753 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.545714 14753 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:35.546113 14753 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.546375 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.548071 13854 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.548381 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.548799 13886 raft_consensus.cc:1238] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:35.548864 14750 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.549368 14750 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.549516 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.549760 14750 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.550413 14737 consensus_queue.cc:1046] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:35.551419 14737 raft_consensus.cc:3053] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.551723 14737 raft_consensus.cc:738] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:35.552500 14737 consensus_queue.cc:260] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.553264 14753 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.553972 14737 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.555233 14753 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.558288 13956 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:35.559892 14745 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:35.561280 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.562136 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.528699 14794 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:35.564740 13854 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.538794 14793 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.603058 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.568550 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.605080 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.561190 14788 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.603914 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.562961 14783 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.567823 14753 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.651234 14764 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.651616 14753 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.651854 14764 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.652089 14753 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.652379 14764 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.568845 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.565392 14786 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.566497 14794 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.602942 13887 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:35.639586 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.568270 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.569121 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.662676 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.630788 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:35.602443 14750 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.677295 13925 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.665509 14757 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:35.667129 14744 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:35.699982 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:35.683984 14746 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:35.732226 14795 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.732831 14753 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:35.733696 14798 log.cc:927] Time spent T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20250623 01:58:35.734182 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:35.736094 14750 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.725524 14746 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.736815 14774 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.721462 14767 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.746507 14753 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.744524 14746 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.749773 14774 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:35.749845 14781 raft_consensus.cc:3053] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.757212 14781 raft_consensus.cc:738] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:35.758107 14781 consensus_queue.cc:260] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.759331 14781 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.726310 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
W20250623 01:58:35.760051 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:35.761741 14511 raft_consensus.cc:2391] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:35.761865 14513 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:35.762071 13808 raft_consensus.cc:1273] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:35.766361 14692 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:35.770503 14745 consensus_queue.cc:1035] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:58:35.774294 14769 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.808913 13953 raft_consensus.cc:2466] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:35.813616 13879 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:35.813779 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.813537 14793 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.815740 14793 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.823132 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.830739 13810 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:35.781051 14732 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:35.835937 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 332940us
I20250623 01:58:35.836114 13852 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:35.837301 13779 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.801697 14772 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.838655 13779 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.838990 14744 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.819974 14743 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.797904 14760 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.839208 14767 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.854638 14786 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:35.862128 13779 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.867071 13852 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:35.874734 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 334463us
I20250623 01:58:35.875564 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.879050 14769 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:35.879519 14513 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 LEARNER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:35.881028 14769 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:35.845896 14764 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.884250 14794 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.886108 14732 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:35.886535 14753 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:35.887281 14753 raft_consensus.cc:695] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:35.888178 14753 consensus_queue.cc:237] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.885702 14764 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.877760 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.866209 14733 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.897305 13883 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:35.898119 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.898803 14794 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.899293 13779 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.899948 14698 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.900369 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.900498 14744 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.900836 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.901160 14698 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.901458 13807 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.878082 14786 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.903594 13779 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.905115 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.908126 13779 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.910657 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.912719 14733 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.913945 13879 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250623 01:58:35.915982 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:35.916288 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:35.917455 13888 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:35.917711 13807 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:35.910043 13779 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.893625 14795 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:35.924350 13779 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.918617 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:35.922741 13776 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
W20250623 01:58:35.930629 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:35.893832 14802 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.938123 13888 raft_consensus.cc:1238] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
W20250623 01:58:35.920282 13925 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 1.500s (SENT) [suppressed 2 similar messages]
I20250623 01:58:35.941720 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.944496 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.942338 14759 consensus_queue.cc:1046] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250623 01:58:35.946511 14759 raft_consensus.cc:3053] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.946841 14759 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:35.926865 14744 consensus_queue.cc:1046] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250623 01:58:35.947763 14744 raft_consensus.cc:3053] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:35.919600 14789 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.947666 14759 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.921962 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:35.922189 14733 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.941180 14790 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:35.944298 14748 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:35.916020 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.950664 14737 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.941439 14775 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.948272 14744 raft_consensus.cc:738] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:35.960142 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:35.956459 13779 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:35.967803 14770 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.955112 14789 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:35.996027 13883 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
W20250623 01:58:35.996397 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 431680us
I20250623 01:58:35.943176 14728 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250623 01:58:35.963811 14759 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.989249 14481 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.002159 14744 consensus_queue.cc:260] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.968631 14748 raft_consensus.cc:695] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:36.003355 14744 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:35.963521 14790 raft_consensus.cc:695] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.002374 14802 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:35.999840 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.006214 14728 raft_consensus.cc:3053] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
W20250623 01:58:36.006875 14805 log.cc:927] Time spent T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20250623 01:58:36.007223 14743 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:36.008924 14786 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:36.010025 14810 log.cc:927] Time spent T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20250623 01:58:36.010299 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.010244 14790 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:36.012269 14746 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:35.999696 14772 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:36.015473 13879 raft_consensus.cc:1238] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.005060 14760 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:36.015915 14738 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:36.017520 13854 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.016395 14739 consensus_queue.cc:1046] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:36.016896 14728 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.004827 14511 raft_consensus.cc:1238] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.028499 13779 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.003898 14748 consensus_queue.cc:237] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.036661 13854 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.038062 13854 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.039322 13854 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.040340 14791 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:36.040760 14791 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:36.041097 14791 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.050673 13854 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.052137 13854 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.032152 14738 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:36.055377 13779 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.057109 13854 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.058437 13779 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.058655 13854 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.055999 14783 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.059612 14785 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.043239 14772 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:36.068775 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:36.062759 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:36.053840 14781 consensus_queue.cc:1046] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250623 01:58:36.066713 14737 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.066715 14808 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.061347 14775 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.043543 14745 raft_consensus.cc:2802] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:36.070013 13879 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.068488 13883 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:36.095755 13779 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.067938 14728 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.094244 14809 consensus_queue.cc:1046] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250623 01:58:36.097028 14728 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.069017 14772 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:36.043193 14760 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:36.061565 14738 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.060658 14733 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.080538 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.081334 14781 raft_consensus.cc:3053] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.065286 14788 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.069242 14793 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.075853 14786 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:36.125725 14781 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.100457 14770 raft_consensus.cc:3053] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.123206 13887 raft_consensus.cc:1238] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.129824 14777 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.100025 14698 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.098235 14785 raft_consensus.cc:3053] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.100486 14772 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.123379 14746 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:36.121615 13779 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.099987 14745 raft_consensus.cc:695] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:36.123487 14760 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:36.134333 14736 consensus_queue.cc:1046] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20250623 01:58:36.158097 13779 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:36.168998 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:36.136632 14781 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.063376 13854 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.137233 14770 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.174382 14743 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:36.183162 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 119883us
I20250623 01:58:36.183885 14789 raft_consensus.cc:3053] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
W20250623 01:58:36.184021 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:36.183714 14788 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.184293 13779 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.184303 14789 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.184700 14775 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.185338 14775 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.185745 13779 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.183928 13854 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.185250 14789 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.174491 14760 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.173147 14733 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.188153 13854 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.140175 14785 raft_consensus.cc:738] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:58:36.181457 14812 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.066s user 0.004s sys 0.000s
I20250623 01:58:36.204738 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.199286 13710 catalog_manager.cc:5582] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 2, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:36.193284 13779 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.205963 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.209323 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.210062 13954 raft_consensus.cc:1238] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.209970 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.209436 13953 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
W20250623 01:58:36.211098 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 290845us
I20250623 01:58:36.211941 13779 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:36.216883 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:36.216913 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:36.193558 14788 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.218163 13779 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.198527 14737 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.206876 14781 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.219998 14770 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.220042 13779 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.207223 14793 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.220074 13925 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.187150 14743 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.215086 14698 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.197440 14759 consensus_queue.cc:1046] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250623 01:58:36.230767 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.231022 13925 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.219669 14733 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.236021 14759 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.236464 14732 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.236692 14759 raft_consensus.cc:3053] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.235670 14785 consensus_queue.cc:260] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:36.237061 14759 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.237649 14785 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.237840 14759 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.238976 14759 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.241677 13925 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.221426 14789 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.222847 14739 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.244724 13925 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.245779 13925 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.250438 14741 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.257429 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.234540 14804 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250623 01:58:36.223433 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 195692us
I20250623 01:58:36.221149 14736 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
W20250623 01:58:36.229822 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 267452us
I20250623 01:58:36.263988 13925 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.265973 13925 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.266814 13925 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:58:36.268164 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:36.246394 14732 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.276875 13709 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:36.273279 14791 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:36.267393 14741 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.250888 14769 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:36.221246 14745 consensus_queue.cc:237] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:36.278664 14816 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:36.266755 14804 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:36.276753 14760 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.269598 14770 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.304224 14513 raft_consensus.cc:1238] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.269487 14806 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.286800 14769 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:36.313388 14769 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.321640 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.284483 14692 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:36.309917 14760 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:36.312191 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.322788 14692 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:36.323175 14692 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.327085 14772 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.327885 14753 consensus_queue.cc:1046] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250623 01:58:36.328713 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:36.329044 14753 raft_consensus.cc:3053] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.329406 14753 raft_consensus.cc:738] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:36.330909 14753 consensus_queue.cc:260] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:36.332372 14753 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.334184 14513 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:36.302074 14816 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:36.296939 14804 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.306638 14738 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.344275 13884 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.344733 14738 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:36.354004 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 134031us
I20250623 01:58:36.354686 14743 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:36.355003 13779 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.355643 13959 raft_consensus.cc:1238] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.355931 14733 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.356526 14816 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:36.358011 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:36.358255 14791 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:36.358644 13883 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:36.360476 13926 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.349741 14772 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:36.379142 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.385372 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.366997 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:36.386792 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 100106us
I20250623 01:58:36.367789 14743 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:58:36.395144 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:36.401888 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:36.388726 13925 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.408767 13926 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:36.368388 14733 consensus_queue.cc:1046] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250623 01:58:36.410699 13925 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.412745 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:36.415689 13810 raft_consensus.cc:2391] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:36.418557 13807 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:36.419413 13804 raft_consensus.cc:1238] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.426195 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.427215 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.427975 13804 raft_consensus.cc:2391] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:36.428759 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.419706 13807 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:36.381914 14478 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:36.429774 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 59750.9us
I20250623 01:58:36.430014 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.431031 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:36.419795 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.421701 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.423693 13806 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250623 01:58:36.429857 14816 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:36.436719 14816 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:36.437196 14816 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.437242 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.420886 13809 raft_consensus.cc:1238] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.413290 14776 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.448922 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.457425 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.458245 13957 raft_consensus.cc:1238] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.459241 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.459267 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.460352 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.463238 13960 raft_consensus.cc:1238] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.448503 14741 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.465353 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.466790 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.466909 14698 raft_consensus.cc:3053] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.467939 13887 raft_consensus.cc:1238] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.468210 13813 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.468688 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.472467 14692 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.477821 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.478817 14769 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.518043 13955 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.451150 14825 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:36.518628 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.518843 14825 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:36.519263 14825 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.499476 13879 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.507004 13958 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.521447 13884 raft_consensus.cc:1238] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
W20250623 01:58:36.521920 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:36.513868 14692 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:36.514482 13953 raft_consensus.cc:2391] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:36.514950 13886 raft_consensus.cc:1238] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.515730 13881 raft_consensus.cc:1238] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.502476 14698 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.519672 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.523370 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:36.526600 13854 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.523988 13959 raft_consensus.cc:3053] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.530009 13854 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.528210 13959 raft_consensus.cc:738] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:36.532203 13854 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.537856 13854 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.541375 14767 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.540959 13959 consensus_queue.cc:260] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.518659 14769 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:36.543509 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 132870us
I20250623 01:58:36.544328 13925 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.547029 13925 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.523944 14804 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.550714 13959 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.550993 14746 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.550654 14741 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.561925 14777 consensus_queue.cc:1046] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:36.562995 14777 raft_consensus.cc:3053] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.563320 14777 raft_consensus.cc:738] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.564113 14777 consensus_queue.cc:260] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.550356 13925 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.551357 14767 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.533082 14791 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.551710 14744 consensus_queue.cc:1046] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:36.552953 14772 consensus_queue.cc:1046] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:36.553126 13854 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.531338 14823 consensus_queue.cc:1046] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250623 01:58:36.541996 14785 consensus_queue.cc:1046] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20250623 01:58:36.542585 14738 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.555734 14759 consensus_queue.cc:1046] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:36.549960 14739 consensus_queue.cc:1046] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20250623 01:58:36.525731 14698 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.565207 14777 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.575558 13959 raft_consensus.cc:1273] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:36.575942 13925 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:36.576563 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 167898us
I20250623 01:58:36.577033 13925 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.579586 14744 raft_consensus.cc:3053] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.580467 14775 raft_consensus.cc:3053] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.580965 14775 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.581069 14698 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.579989 14744 raft_consensus.cc:738] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.581743 14775 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.575135 14770 raft_consensus.cc:3053] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.583468 14785 raft_consensus.cc:3053] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.589238 14785 raft_consensus.cc:738] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:36.584275 14772 raft_consensus.cc:3053] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.582427 14741 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.586431 14744 consensus_queue.cc:260] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.588183 14831 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:36.588377 14770 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.588577 14823 raft_consensus.cc:3053] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.591203 13925 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.592736 14772 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:36.597229 14743 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:36.595422 14732 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250623 01:58:36.595721 13962 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:36.600876 14770 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
W20250623 01:58:36.596553 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 43538.2us
I20250623 01:58:36.596763 14753 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.597648 14772 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.598264 14823 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.595356 14746 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250623 01:58:36.606079 13925 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.610209 13925 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.590175 14785 consensus_queue.cc:260] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.608703 13955 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.608891 14772 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.606010 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.601881 14770 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.612304 13925 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.617195 14831 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.617275 14753 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.595091 14825 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.619206 14825 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:36.619824 13956 raft_consensus.cc:1238] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:36.611991 14776 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.625882 14804 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:36.609937 14775 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.611331 14743 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.598145 14741 consensus_queue.cc:1046] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20250623 01:58:36.608749 14744 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.626699 14823 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.635389 14823 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.636813 14513 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:36.639799 14739 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:58:36.642428 13779 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.644929 13779 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.645565 14744 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.645998 14743 raft_consensus.cc:3053] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.646956 14791 consensus_queue.cc:1046] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:36.648443 13854 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.650051 14733 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.651278 14743 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:36.657230 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.649268 14739 consensus_queue.cc:1046] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20250623 01:58:36.659210 14739 raft_consensus.cc:3053] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.659621 14739 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:36.649637 14830 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.661631 14738 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.663384 13854 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.664064 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.665172 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.665815 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.654459 14785 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.673419 14692 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.678388 14816 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:36.680737 14743 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.682175 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.682569 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:36.683058 14743 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.683267 13809 raft_consensus.cc:2391] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:36.684769 14738 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.673207 14767 raft_consensus.cc:3053] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.686131 14791 consensus_queue.cc:1046] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:36.686206 13854 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.686460 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:36.661707 14739 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.688561 14738 raft_consensus.cc:3053] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.691869 14738 raft_consensus.cc:738] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:36.688413 13854 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.687418 14753 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.672587 14772 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.694056 13886 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:36.692665 14738 consensus_queue.cc:260] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.696622 14733 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:36.697470 14753 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.681401 14816 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:36.685671 14767 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:36.702497 13854 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:58:36.714136 14832 log.cc:927] Time spent T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250623 01:58:36.697300 14738 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.736584 13810 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:36.742053 13854 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.735843 14772 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.749226 13854 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.750735 14760 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.740254 14764 consensus_queue.cc:1046] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20250623 01:58:36.764780 13711 catalog_manager.cc:5582] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 2, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 2 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:36.767668 14788 raft_consensus.cc:2747] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.767212 13854 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.773977 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:36.775185 14793 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.756276 14767 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.779316 13926 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:36.787748 13808 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:36.788266 14739 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.787801 14767 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:36.790035 13926 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:36.776160 13881 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:36.777822 14764 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.796929 14830 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:36.804790 14692 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:36.826272 13956 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250623 01:58:36.826933 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 215015us
I20250623 01:58:36.827967 13925 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.846115 13925 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.802012 14788 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250623 01:58:36.811645 14793 consensus_queue.cc:1046] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250623 01:58:36.830763 14830 raft_consensus.cc:695] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:36.888162 13925 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.833946 14804 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.835822 14692 raft_consensus.cc:695] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:36.890851 13925 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.842913 14748 raft_consensus.cc:3053] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:36.896457 13810 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:36.847187 13957 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:36.845585 14745 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:36.884253 14769 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.893877 14732 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.913995 13925 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.843401 14478 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:36.922662 13925 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:36.930794 13925 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:36.938230 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 106740us
I20250623 01:58:36.900722 14748 raft_consensus.cc:738] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:58:36.951968 14835 log.cc:927] Time spent T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20250623 01:58:36.890190 14821 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20250623 01:58:36.962492 14732 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:36.963500 13813 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:36.923751 14830 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.913061 14808 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20250623 01:58:36.921247 14836 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.978840 13808 raft_consensus.cc:1273] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:36.920678 14692 consensus_queue.cc:237] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:36.910519 14745 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:36.949060 14769 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:36.996439 14741 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.014431 13710 catalog_manager.cc:5582] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 2, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:36.996573 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 65890.8us
I20250623 01:58:36.928205 14804 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.007890 13959 raft_consensus.cc:1238] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:37.021481 13925 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.021456 13710 catalog_manager.cc:5582] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:37.025398 13925 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.031776 14803 raft_consensus.cc:3053] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:37.032204 14803 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:37.036741 13925 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.038388 13925 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.039559 13925 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.040657 13925 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.028158 14806 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.048574 14791 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
W20250623 01:58:37.027477 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 76765.1us
I20250623 01:58:37.052790 14844 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:37.041146 14793 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20250623 01:58:37.041049 14776 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.041251 14804 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.060391 13925 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.048807 14769 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.048664 14741 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.051105 14825 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.032967 14803 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:37.163388 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 98466.8us
I20250623 01:58:37.041568 14748 consensus_queue.cc:260] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:37.246718 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 70756.7us
I20250623 01:58:37.130193 14830 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.120373 14789 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:37.253962 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 193725us
I20250623 01:58:37.259606 14836 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250623 01:58:37.250015 14808 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:37.267985 13925 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.273653 13925 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.274646 14825 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.304724 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:37.256667 14789 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:37.274896 14775 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.273138 14854 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:37.312680 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.285333 14806 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.309139 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:37.313758 13883 raft_consensus.cc:2391] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:37.315244 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:37.270417 14830 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20250623 01:58:37.345803 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:37.272877 14808 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.347942 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:37.285691 14769 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250623 01:58:37.351171 13888 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:37.349716 13883 raft_consensus.cc:2391] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:58:37.347965 14837 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20250623 01:58:37.318171 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:37.356541 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:37.318517 14745 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.355931 13925 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.359819 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:37.363597 13925 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.318611 14775 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.358395 14849 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.359745 14826 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.372345 14855 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.372823 14855 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:37.373862 13805 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:37.361428 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.374763 14855 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.368212 14743 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:37.361856 14745 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:37.318557 14854 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:37.382547 13880 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:37.386600 13925 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:37.387252 14804 consensus_queue.cc:1046] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:37.388626 14741 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:37.388800 13925 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:37.389016 14741 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:37.389552 14853 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:37.390022 14853 raft_consensus.cc:695] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:37.389811 14741 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.390362 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:37.391105 14741 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:37.390851 14853 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.395254 14808 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.397521 14789 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:37.385468 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:37.391033 13811 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
W20250623 01:58:37.407503 14850 log.cc:927] Time spent T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20250623 01:58:37.408896 13811 raft_consensus.cc:1273] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:37.410274 14479 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:58:37.409817 14510 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:37.411885 14816 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:37.412318 14816 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:37.412746 14816 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.413939 14857 consensus_queue.cc:1035] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:37.381856 14849 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:37.417665 14849 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:37.418318 14792 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.397s user 0.003s sys 0.000s
I20250623 01:58:37.383850 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.419416 13888 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:37.415474 14745 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:37.365185 14844 raft_consensus.cc:695] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:37.422333 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:37.403010 13962 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:37.388206 14804 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.425073 13886 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
W20250623 01:58:37.426185 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:37.452764 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:37.257586 14783 consensus_queue.cc:1162] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c243715fe12445019c33bbeafbaea635" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250623 01:58:37.469832 14478 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:37.470563 14854 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.471696 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:37.473270 14844 consensus_queue.cc:237] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:37.474889 14775 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.476930 14745 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.477478 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:37.403436 14826 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.480719 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:37.498502 14789 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.499053 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:37.498991 14789 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.499816 13882 raft_consensus.cc:2391] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:37.500921 14789 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.502514 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:37.504163 14849 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:37.504690 14849 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:37.505121 14849 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.510032 14481 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:37.510885 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:37.512439 14745 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.513438 13880 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:58:37.525874 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 101131us
I20250623 01:58:37.527853 13925 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:37.534534 14806 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:37.476943 14511 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:37.542069 14806 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:37.469170 13884 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:37.493520 14826 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.526026 14698 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.562079 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.562744 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:37.562907 13883 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:37.531123 14743 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.566483 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.571894 13883 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:37.575250 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.576084 13888 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:37.570551 14698 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.570837 14732 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.580250 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.580591 14698 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.580868 13886 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:37.534549 14816 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:37.582917 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.583747 13879 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:37.543213 14806 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.575572 14823 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.549417 14479 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:37.593287 14823 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.597718 13925 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:37.605000 13776 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:37.593797 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 62804.8us
I20250623 01:58:37.609162 13776 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:37.615034 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:37.614006 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.607438 14743 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:37.622810 14743 raft_consensus.cc:695] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:37.623584 13887 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:37.596199 14478 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:37.597450 14842 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:37.627115 14741 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:37.627961 14844 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:37.638765 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:37.663411 13778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 238999us
I20250623 01:58:37.611788 14789 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:37.668406 13852 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:37.673552 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.674301 13886 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:37.675523 13776 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:37.616948 14775 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:37.639725 14844 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:37.636438 14698 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:37.682165 13879 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:37.681972 14512 raft_consensus.cc:2391] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:37.673555 14789 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:37.623657 14743 consensus_queue.cc:237] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:37.717075 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 48789.4us
I20250623 01:58:37.724973 14770 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:37.725538 14770 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:37.725948 14770 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.685688 14775 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:37.703122 14698 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:37.698437 14808 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.697765 14844 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.695221 14826 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.704864 14823 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.741045 13882 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:37.739144 14698 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.645457 14842 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:58:37.707621 14789 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.739706 14775 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.739495 14846 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:37.808153 14826 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.857872 14826 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.808517 13886 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:37.739737 14866 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:37.898806 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 223212us
I20250623 01:58:37.915473 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:37.916211 13886 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:37.914395 13776 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:37.925971 13886 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:37.903820 14816 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.888691 14853 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20250623 01:58:37.914608 14854 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:37.937990 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:37.938557 13959 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.931095 14826 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:37.633229 14831 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:58:37.947954 14868 log.cc:927] Time spent T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
W20250623 01:58:37.954610 14867 log.cc:927] Time spent T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.150s user 0.000s sys 0.001s
I20250623 01:58:37.953096 13959 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:37.930083 14823 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:37.903975 14866 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:37.958101 14826 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.967208 14866 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:37.967932 14866 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:37.968729 14826 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:37.968436 14866 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.933770 14804 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20250623 01:58:37.960588 14883 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:37.969732 14883 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:37.970090 14883 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:37.976193 14481 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:37.884902 14846 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.952593 14770 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:37.966069 14698 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:37.995664 14860 log.cc:927] Time spent T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.289s user 0.001s sys 0.000s
I20250623 01:58:37.992883 14760 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:38.003960 14821 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:37.978134 14823 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:38.029428 14849 consensus_meta.cc:287] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: Time spent flushing consensus metadata: real 0.524s user 0.004s sys 0.007s
I20250623 01:58:38.036446 14849 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.039460 14849 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.040066 14789 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.026703 14821 raft_consensus.cc:695] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:38.021134 14760 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.048590 14698 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:38.035296 14848 log.cc:927] Time spent T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.728s user 0.006s sys 0.001s
I20250623 01:58:38.052265 14883 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.052839 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:38.053525 14853 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:38.042171 14846 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.035580 14842 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.071838 13957 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
W20250623 01:58:38.081522 14873 log.cc:927] Time spent T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250623 01:58:38.081923 14823 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.074396 14853 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:38.084416 14853 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:38.087150 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:38.088160 13888 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:38.093400 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 58) took 1537 ms. Trace:
I20250623 01:58:38.099633 14883 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.145579 14836 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:38.096748 14698 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:38.159710 13882 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:38.160256 14789 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.155407 14770 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.093779 13711 rpcz_store.cc:276] 0623 01:58:36.556473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:36.556654 (+ 181us) service_pool.cc:225] Handling call
0623 01:58:36.759593 (+202939us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54003868d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e97bafac 00007f41e98b37ee 00007f41e7dfd624 00007f41e7dfd5b2 00007f41e7dfd4e1 00007f41e7dfd45c 00007f41e7dfc712 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:38.093333 (+1333740us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:36.787530 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:36.788080 (+ 550us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:36.788112 (+ 32us) write_op.cc:435] Acquired schema lock
0623 01:58:36.788127 (+ 15us) tablet.cc:660] Decoding operations
0623 01:58:36.788492 (+ 365us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:36.788540 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:36.788569 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:58:36.788746 (+ 177us) tablet.cc:699] Row locks acquired
0623 01:58:36.788764 (+ 18us) write_op.cc:260] PREPARE: finished
0623 01:58:36.788903 (+ 139us) write_op.cc:270] Start()
0623 01:58:36.789014 (+ 111us) write_op.cc:276] Timestamp: P: 1750643916788860 usec, L: 0
0623 01:58:36.789033 (+ 19us) op_driver.cc:348] REPLICATION: starting
0623 01:58:36.923348 (+134315us) log.cc:844] Serialized 529 byte log entry
0623 01:58:36.925166 (+ 1818us) op_driver.cc:464] REPLICATION: finished
0623 01:58:36.956302 (+ 31136us) write_op.cc:301] APPLY: starting
0623 01:58:36.956393 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:36.956531 (+ 138us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:36.956665 (+ 134us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:36.956932 (+ 267us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:36.957125 (+ 193us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:58:36.957184 (+ 59us) write_op.cc:312] APPLY: finished
0623 01:58:36.987515 (+ 30331us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54003868d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9aa18 00007f41e74c0811 00007f41e74bffe8 00007f41e74c4c81 00007f41e74c4c39 00007f41e74c4bc9 00007f41e74c4b91 00007f41e74c3dbd 00007f41e9b9b124 00007f41e9b9af59
0623 01:58:37.000485 (+ 12970us) log.cc:844] Serialized 22 byte log entry
0623 01:58:37.014684 (+ 14199us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:37.019544 (+ 4860us) write_op.cc:454] Released schema lock
0623 01:58:37.020542 (+ 998us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:37.063388 (+ 42846us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:37.066762 (+ 3374us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":410103168,"child_traces":[["op",{"apply.queue_time_us":30871,"apply.run_cpu_time_us":1903,"apply.run_wall_time_us":79563,"num_ops":1,"prepare.queue_time_us":1604,"prepare.run_cpu_time_us":5454,"prepare.run_wall_time_us":208326,"replication_time_us":135989,"spinlock_wait_cycles":300795136,"thread_start_us":75462,"threads_started":1}]]}
I20250623 01:58:38.163810 13810 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:38.152663 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:38.172611 14853 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250623 01:58:38.145215 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 47784.7us
I20250623 01:58:38.096814 14775 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.171228 14804 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:38.153093 14836 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.180002 13804 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.195930 13882 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:38.179255 14866 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.174175 14698 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:38.200987 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 288081us
I20250623 01:58:38.200636 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:38.152071 14821 consensus_queue.cc:237] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.209218 13888 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.211738 13882 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:38.217440 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:38.218042 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:38.218974 13955 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
W20250623 01:58:38.228538 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:38.237644 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:38.195858 14861 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.218048 13958 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:38.197615 14760 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.196493 14804 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:38.256386 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:38.227365 14866 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.203842 14816 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20250623 01:58:38.251801 14861 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.279477 14861 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:38.272650 13881 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.968s user 0.006s sys 0.021s
I20250623 01:58:38.253225 14806 consensus_queue.cc:1046] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250623 01:58:38.263185 14853 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:38.267379 14743 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:38.273692 14857 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20250623 01:58:38.272409 14842 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.302409 14804 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:38.302953 14743 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:38.283110 13881 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.993s user 0.006s sys 0.021s
W20250623 01:58:38.330371 13805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 423) took 1674 ms (client timeout 1864 ms). Trace:
W20250623 01:58:38.331002 13805 rpcz_store.cc:269] 0623 01:58:36.655612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:36.657694 (+ 2082us) service_pool.cc:225] Handling call
0623 01:58:38.330086 (+1672392us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b5400153bd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:38.330246 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":760591104}
I20250623 01:58:38.317831 14902 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:38.335075 14902 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.295733 14853 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.306741 14844 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.347769 14743 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:38.348469 14844 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.350641 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:38.351370 13886 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.324088 14836 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:38.327594 14775 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.271600 14866 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:38.360427 14902 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:38.362464 14698 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.321703 14842 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.368678 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:38.369073 14872 log.cc:927] Time spent T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.457s user 0.000s sys 0.003s
I20250623 01:58:38.370525 13958 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
W20250623 01:58:38.387413 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 473084us
I20250623 01:58:38.395474 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:38.396234 13883 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.398397 13888 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:38.400710 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:38.401705 14478 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:38.385824 14853 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:38.434379 13886 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
W20250623 01:58:38.473639 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 72008.7us
I20250623 01:58:38.478902 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:38.479812 13886 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:38.484763 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:38.485491 13809 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:38.497117 13812 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:38.497640 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:38.498879 13886 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.519369 14887 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:38.528364 13776 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:38.527256 14823 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:38.533980 14823 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.534797 14842 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.534862 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:38.520728 14789 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:38.535828 14789 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.536914 14823 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:38.520031 14887 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.535616 13884 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.540551 13812 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:38.540316 14738 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.544183 14893 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:38.544310 14738 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.487751 14770 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:38.544833 14893 raft_consensus.cc:695] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:38.495015 14775 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:38.494292 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 90881.3us
I20250623 01:58:38.521034 14883 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:38.538565 14882 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.527472 14739 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:38.521512 14853 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:38.553131 14698 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:38.559571 14879 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:38.560216 14738 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:38.554368 14842 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.571991 14877 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:38.575567 14877 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.532997 13958 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:38.561123 14775 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:38.563069 14890 log.cc:927] Time spent T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.187s user 0.000s sys 0.002s
I20250623 01:58:38.563143 14883 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.572831 14698 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:38.555521 13883 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:38.569962 13957 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:38.571396 14875 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:38.572400 14789 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.567276 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:38.584875 14775 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.558588 14770 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.593083 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:38.600135 13888 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.601963 13883 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:38.589828 14698 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.583640 14823 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.604722 14775 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.603093 14789 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.610230 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:38.572599 14739 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.576823 14842 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.574573 14853 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.607916 13884 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:38.648125 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:38.648761 14842 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.648931 13884 raft_consensus.cc:2391] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:38.614951 13883 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.649286 14739 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:38.638137 14789 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.650048 14739 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.650506 14739 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.652171 14789 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.652809 14739 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.653012 13810 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:38.644454 13888 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:38.646602 14887 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.657733 13883 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:38.611442 14883 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.609815 14511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48412 (request call id 39) took 1206 ms. Trace:
I20250623 01:58:38.611655 13804 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:38.607523 14893 consensus_queue.cc:237] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.582907 14882 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.615906 14775 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.655136 14481 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:58:38.689674 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 74926.8us
I20250623 01:58:38.689836 14877 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.690248 13806 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:38.614392 14823 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.616451 14875 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:38.580933 14879 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.681843 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:38.702103 13806 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:38.721598 13884 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:38.694664 14847 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:38.678110 14511 rpcz_store.cc:276] 0623 01:58:37.407689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:37.414364 (+ 6675us) service_pool.cc:225] Handling call
0623 01:58:37.414668 (+ 304us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:37.476825 (+ 62157us) raft_consensus.cc:1532] Early marking committed up to index 3
0623 01:58:37.620860 (+144035us) log.cc:844] Serialized 14 byte log entry
0623 01:58:37.903068 (+282208us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:37.908198 (+ 5130us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:37.908583 (+ 385us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:38.609051 (+700468us) spinlock_profiling.cc:243] Waited 61.9 ms on lock 0x7b540056dbd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:38.609099 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:38.609245 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2206,"raft.run_cpu_time_us":1076,"raft.run_wall_time_us":10060,"spinlock_wait_cycles":142389504,"thread_start_us":268932,"threads_started":1,"wal-append.queue_time_us":378443}
I20250623 01:58:38.680758 13886 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:38.714179 14770 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.746161 14879 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.751941 14866 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:38.753716 13887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 408) took 1104 ms. Trace:
I20250623 01:58:38.754264 13887 rpcz_store.cc:276] 0623 01:58:37.633692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:37.636828 (+ 3136us) service_pool.cc:225] Handling call
0623 01:58:37.700168 (+ 63340us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:37.923172 (+223004us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:37.926677 (+ 3505us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:38.009188 (+ 82511us) log.cc:844] Serialized 548 byte log entry
0623 01:58:38.272820 (+263632us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:38.276289 (+ 3469us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:38.715724 (+439435us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54001a4d54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:38.716654 (+ 930us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:38.717382 (+ 728us) raft_consensus.cc:1683] finished
0623 01:58:38.717406 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:38.736132 (+ 18726us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436293504,"thread_start_us":250048,"threads_started":1,"wal-append.queue_time_us":362735}
I20250623 01:58:38.756685 14866 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.757591 14866 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.758363 14775 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.760943 14866 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.764384 14748 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
W20250623 01:58:38.772099 13854 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.023s sys 0.029s
W20250623 01:58:38.772541 13854 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.023s sys 0.029s
I20250623 01:58:38.694840 14853 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:38.794072 13882 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.020s sys 0.072s
W20250623 01:58:38.794541 13882 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s user 0.020s sys 0.072s
I20250623 01:58:38.803875 14882 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.810034 13879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 409) took 1069 ms. Trace:
I20250623 01:58:38.810619 13879 rpcz_store.cc:276] 0623 01:58:37.740449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:37.744260 (+ 3811us) service_pool.cc:225] Handling call
0623 01:58:37.887708 (+143448us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:37.916489 (+ 28781us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:37.919945 (+ 3456us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:37.984557 (+ 64612us) log.cc:844] Serialized 26 byte log entry
0623 01:58:38.395929 (+411372us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:38.396081 (+ 152us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:38.396463 (+ 382us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:38.809743 (+413280us) raft_consensus.cc:1683] finished
0623 01:58:38.809777 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:38.809957 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6389,"raft.queue_time_us":11267,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":13651,"thread_start_us":357290,"threads_started":1,"wal-append.queue_time_us":792741}
W20250623 01:58:38.819308 13927 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.027s sys 0.039s
W20250623 01:58:38.819842 13927 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.584s user 0.027s sys 0.039s
I20250623 01:58:38.825946 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:38.816941 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:38.807665 14698 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.856839 14846 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:38.840266 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:38.857438 14846 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:38.865111 13925 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.008s sys 0.018s
W20250623 01:58:38.866119 13925 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.009s sys 0.018s
W20250623 01:58:38.870446 13926 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.005s sys 0.022s
W20250623 01:58:38.871454 13926 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.005s sys 0.022s
W20250623 01:58:38.870438 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 57142.5us
W20250623 01:58:38.877462 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 616558us
I20250623 01:58:38.883320 13854 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:38.891911 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:38.892727 13958 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
W20250623 01:58:38.865480 14821 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.014s sys 0.014s
W20250623 01:58:38.873961 14806 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.006s sys 0.020s
I20250623 01:58:38.889097 13926 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:38.872500 14853 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:38.894929 13881 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.017s sys 0.018s
I20250623 01:58:38.874356 14741 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:58:38.896348 14821 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s user 0.014s sys 0.014s
W20250623 01:58:38.896534 14806 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.575s user 0.006s sys 0.020s
I20250623 01:58:38.887354 14794 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250623 01:58:38.901677 13881 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s user 0.018s sys 0.018s
I20250623 01:58:38.899519 14823 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:38.901486 14741 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.916019 14794 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:38.903149 14853 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.940306 14897 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:38.957580 14794 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
W20250623 01:58:38.958716 14831 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.003s sys 0.049s
I20250623 01:58:38.978674 14732 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
W20250623 01:58:39.028138 13853 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.018s sys 0.030s
W20250623 01:58:39.029891 13853 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.018s sys 0.030s
I20250623 01:58:39.032604 13776 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:39.033854 14882 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:39.034360 14882 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:39.034775 14882 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:38.974933 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 944) took 1913 ms. Trace:
I20250623 01:58:38.973356 14910 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.046949 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:39.057631 13956 raft_consensus.cc:1238] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:39.058682 13955 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:38.939177 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 68813.9us
I20250623 01:58:39.062357 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:39.028699 14831 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.004s sys 0.049s
I20250623 01:58:38.974267 14897 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:38.978454 14875 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:39.020031 14698 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.027787 14732 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:39.054587 14785 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:39.102103 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:39.107895 13851 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.003s sys 0.046s
I20250623 01:58:39.109486 13854 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:39.117511 13961 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:39.043752 14847 raft_consensus.cc:695] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
W20250623 01:58:39.112020 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 222931us
W20250623 01:58:39.117997 13851 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.690s user 0.003s sys 0.046s
I20250623 01:58:39.054184 14748 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:39.121111 13926 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:58:39.089887 14908 log.cc:927] Time spent T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20250623 01:58:39.062219 14849 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.063252 14866 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:39.074479 14896 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:39.074303 14910 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.195097 13809 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250623 01:58:39.224663 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:39.227661 13812 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:39.211053 13776 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:39.230728 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 109717us
W20250623 01:58:39.229976 13852 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.830s user 0.001s sys 0.026s
W20250623 01:58:39.232193 13852 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.833s user 0.001s sys 0.026s
W20250623 01:58:39.232607 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 833287us
I20250623 01:58:39.078429 14770 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:39.245359 13776 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:39.246718 13776 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:39.090071 14826 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:58:39.196697 14912 log.cc:927] Time spent T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250623 01:58:39.110073 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:39.280781 14923 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:39.281045 13813 raft_consensus.cc:2408] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20250623 01:58:39.226354 14808 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.283460 14932 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.283424 14808 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:39.284054 14932 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.285036 14770 raft_consensus.cc:695] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:39.285509 14732 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.184595 14783 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:39.286504 14783 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.240520 14905 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:39.186710 14767 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.287995 14905 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.288285 14767 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.288883 14783 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.290263 14767 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.133615 14795 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:39.292124 14795 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:39.292590 14795 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:39.282040 13879 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250623 01:58:39.201375 14906 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.281244 14923 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.226296 14853 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:39.188244 14741 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:39.127055 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:39.272272 13776 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:39.156054 14843 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:39.248947 14879 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250623 01:58:39.340065 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 67967.7us
I20250623 01:58:39.278630 14883 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:39.280025 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:39.135887 14785 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.241428 14849 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:39.247398 14866 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.248760 14896 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.159085 14748 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:39.286069 14770 consensus_queue.cc:237] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.346143 13776 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
W20250623 01:58:39.783245 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 437157us
I20250623 01:58:39.784016 13776 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:39.406147 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 174811us
W20250623 01:58:39.487051 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 261039us
I20250623 01:58:39.279292 14698 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:39.329442 14821 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.460s
I20250623 01:58:39.789304 14795 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.355222 14906 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.791389 14849 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:39.791930 14795 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.792476 14906 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.794114 14897 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.794869 14897 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:39.795629 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:39.655845 14924 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.796342 13879 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:39.459448 14917 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:39.796445 14924 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.797303 14917 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.797806 14478 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:39.044486 13881 rpcz_store.cc:276] 0623 01:58:37.061371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:37.064380 (+ 3009us) service_pool.cc:225] Handling call
0623 01:58:37.126071 (+ 61691us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:37.248222 (+122151us) raft_consensus.cc:1532] Early marking committed up to index 5
0623 01:58:37.954624 (+706402us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e0b22e7e 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3d565 00007f41e7482657
0623 01:58:38.327492 (+372868us) log.cc:844] Serialized 14 byte log entry
0623 01:58:38.944041 (+616549us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:38.945700 (+ 1659us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:58:38.957395 (+ 11695us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:38.960018 (+ 2623us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:38.973498 (+ 13480us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1451,"prepare.queue_time_us":769255,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":3769,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":11724,"spinlock_wait_cycles":22990592,"thread_start_us":1615791,"threads_started":2,"wal-append.queue_time_us":637809}
I20250623 01:58:39.391537 14847 consensus_queue.cc:237] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:39.296877 14826 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:39.730530 14913 log.cc:927] Time spent T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.393s user 0.001s sys 0.000s
I20250623 01:58:39.692691 14803 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:39.344877 14808 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:39.789266 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:39.287981 14932 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:39.819917 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:39.820331 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:39.406455 14843 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:39.751804 14931 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:39.344961 14879 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:39.829475 14879 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:39.730806 14858 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:39.829398 14931 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.830205 14858 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.831471 14931 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:39.754915 14882 consensus_meta.cc:287] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.720s user 0.005s sys 0.006s
I20250623 01:58:39.459232 14876 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:39.832136 14882 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.350924 14741 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:39.832762 14876 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.344873 14877 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:39.834079 14882 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.834730 14882 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:39.835095 14882 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:39.835471 14882 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:39.835522 14876 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.642686 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:39.605350 14775 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:39.840098 14775 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.842182 14775 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.842706 14775 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:39.843116 14775 raft_consensus.cc:695] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:39.833132 14741 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:39.752388 14919 log.cc:927] Time spent T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.502s user 0.000s sys 0.000s
I20250623 01:58:39.845486 14741 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:39.845204 14794 consensus_meta.cc:287] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.860s user 0.008s sys 0.000s
I20250623 01:58:39.846055 14741 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:39.329896 14923 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
W20250623 01:58:39.804759 14831 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.003s sys 0.006s
I20250623 01:58:39.828892 14843 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:39.848807 13962 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:39.870622 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:39.818261 14821 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20250623 01:58:39.468045 14853 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:39.869731 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:39.799396 14924 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:39.550823 14875 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:39.755113 14836 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:39.846728 14794 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.795469 14897 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.558363 14916 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:39.801820 14826 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.799561 14917 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:39.845160 14928 log.cc:927] Time spent T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.394s user 0.000s sys 0.001s
W20250623 01:58:39.932874 13927 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.169s sys 0.322s
W20250623 01:58:40.090837 13927 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.840s user 0.191s sys 0.358s
I20250623 01:58:40.091030 14866 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:39.636909 13953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37408 (request call id 352) took 1925 ms. Trace:
I20250623 01:58:39.885651 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.092621 13954 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:40.093008 13955 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
W20250623 01:58:39.967406 13925 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.000s sys 0.013s
W20250623 01:58:40.095755 13925 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.825s user 0.002s sys 0.014s
W20250623 01:58:40.096089 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 845978us
I20250623 01:58:40.098241 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.099004 13958 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:40.099287 13956 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:40.100023 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.100800 13961 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
W20250623 01:58:39.968443 13852 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.723s user 0.005s sys 0.009s
W20250623 01:58:40.101927 13852 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.859s user 0.007s sys 0.010s
W20250623 01:58:40.102340 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 860411us
I20250623 01:58:39.802621 14883 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:40.103526 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.108335 13954 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:40.103935 13852 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:58:39.908644 14831 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.004s sys 0.007s
I20250623 01:58:39.865814 14849 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:39.832747 14858 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:40.119939 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.131208 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.109755 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.130765 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:40.135413 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:40.133496 13777 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.003s sys 0.022s
W20250623 01:58:40.143035 13777 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.614s user 0.003s sys 0.022s
I20250623 01:58:40.140144 13955 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:40.092607 14745 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:40.146754 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.149489 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:40.150178 13954 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:40.144922 13961 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:40.148056 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:40.151542 13958 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:40.092046 13953 rpcz_store.cc:276] 0623 01:58:37.711849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:37.738660 (+ 26811us) service_pool.cc:225] Handling call
0623 01:58:39.636132 (+1897472us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400337554. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e89f509e 00007f41e209f8c7 00007f41e209f852 00007f41e209f781 00007f41e209f6fc 00007f41e209e9b2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:39.636792 (+ 660us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2851515776}
I20250623 01:58:40.094444 14836 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:39.932451 14876 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:40.052282 14748 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:40.133159 14929 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:40.170377 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.159796 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.116597 13957 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250623 01:58:40.104692 14927 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:40.162353 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.186703 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.185009 14931 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:40.133245 14849 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:40.191262 13883 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
W20250623 01:58:40.157051 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 53217.9us
I20250623 01:58:40.103598 14875 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:40.203320 13778 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.022s sys 0.034s
W20250623 01:58:40.203805 13778 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.022s sys 0.034s
I20250623 01:58:40.131693 14877 raft_consensus.cc:695] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:40.199743 13852 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:40.202777 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:40.197741 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 379287us
I20250623 01:58:40.182273 14906 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:40.108409 14775 consensus_queue.cc:237] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:39.866508 14909 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.324s
I20250623 01:58:40.184345 14858 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:40.159014 14785 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:40.157240 14932 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:40.105405 14853 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.154635 14745 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:40.109107 14916 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.215274 13888 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:40.645920 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.295383 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:40.647123 13883 raft_consensus.cc:2408] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:40.229564 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:40.362522 13779 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.004s sys 0.012s
W20250623 01:58:40.649896 13779 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.868s user 0.004s sys 0.028s
W20250623 01:58:40.650220 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 1.36065e+06us
W20250623 01:58:40.651175 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 451520us
W20250623 01:58:40.215284 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 431308us
I20250623 01:58:40.193759 14924 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:40.650375 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 355088us
I20250623 01:58:40.300071 13955 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:40.654896 13854 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:40.659861 13958 consensus_meta.cc:287] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.508s user 0.009s sys 0.000s
I20250623 01:58:40.673418 13958 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:40.669737 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.687194 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.687669 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:40.688704 13956 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:40.689237 13883 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:40.688778 13955 raft_consensus.cc:2391] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:40.204137 14849 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:40.654192 14481 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.016s sys 0.015s
I20250623 01:58:40.692070 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.203037 14910 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:40.268966 14698 raft_consensus.cc:695] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:40.318544 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:40.231105 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:40.702126 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.197031 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.702499 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.609321 14942 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:40.199270 14821 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:58:40.668917 14879 consensus_meta.cc:287] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.839s user 0.020s sys 0.014s
I20250623 01:58:40.279093 14878 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:40.294066 14741 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:40.202632 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.188778 14876 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.312114 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:40.268859 14935 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:40.489847 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:40.659705 13779 connection.cc:232] RPC call timeout handler was delayed by -1.3361191940000481s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
I20250623 01:58:40.214159 14770 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:40.582494 14944 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:40.194828 14905 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:40.657909 14794 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:40.230901 14917 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:40.656818 14932 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.202389 14927 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:40.188396 14929 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.318356 14836 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.214783 14883 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.535023 14748 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:40.508038 14877 consensus_queue.cc:237] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.698458 14906 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:40.583209 14896 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:41.078174 14748 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.055979 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:41.079579 13808 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:41.080077 13808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 1004) took 2017 ms (client timeout 1502 ms). Trace:
I20250623 01:58:41.080058 14906 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:40.656535 14795 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:41.078101 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:40.230360 14866 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:40.711194 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 56359.8us
I20250623 01:58:41.085964 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 960) took 1189 ms. Trace:
I20250623 01:58:41.082206 13883 raft_consensus.cc:2408] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250623 01:58:41.084810 13779 connection.cc:232] RPC call timeout handler was delayed by -1.2716145030000234s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20840000000000003s for a response
I20250623 01:58:40.424571 14930 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:41.086938 13955 rpcz_store.cc:276] 0623 01:58:39.896684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.710918 (+814234us) service_pool.cc:225] Handling call
0623 01:58:41.085910 (+374992us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:41.087241 13925 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 1.873s (SENT) [suppressed 8 similar messages]
W20250623 01:58:41.067369 14940 log.cc:927] Time spent T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.477s user 0.001s sys 0.000s
W20250623 01:58:40.705770 14481 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.017s sys 0.015s
W20250623 01:58:41.033355 14803 consensus_meta.cc:287] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 1.231s user 0.012s sys 0.000s
I20250623 01:58:41.109380 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:41.048792 14950 log.cc:927] Time spent T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
W20250623 01:58:41.167487 13954 consensus_meta.cc:287] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 1.012s user 0.005s sys 0.005s
I20250623 01:58:41.185770 13954 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:41.186337 13954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 966) took 1287 ms. Trace:
I20250623 01:58:41.186373 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:41.186738 13954 rpcz_store.cc:276] 0623 01:58:39.898540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.149198 (+250658us) service_pool.cc:225] Handling call
0623 01:58:41.186271 (+1037073us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:40.686532 14924 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:40.824311 14925 log.cc:927] Time spent T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.666s user 0.000s sys 0.001s
I20250623 01:58:40.816658 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:40.692793 14897 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:41.055035 14745 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:41.080055 14748 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:41.187640 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:41.185792 14478 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.007s sys 0.006s
W20250623 01:58:41.297150 14478 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.648s user 0.012s sys 0.014s
I20250623 01:58:41.297386 13954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 969) took 1363 ms. Trace:
I20250623 01:58:41.170784 14866 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:40.924435 14875 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:40.750890 14910 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:41.187383 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 959) took 1316 ms. Trace:
I20250623 01:58:41.084546 14698 consensus_queue.cc:237] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:41.297855 13851 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.024s sys 0.030s
W20250623 01:58:41.340916 13851 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.600s user 0.025s sys 0.031s
I20250623 01:58:41.340448 14910 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.102496 14795 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:41.044806 14785 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250623 01:58:40.977329 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 253365us
I20250623 01:58:41.056644 14945 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:41.340201 13852 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.027s sys 0.051s
W20250623 01:58:41.408411 13852 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.751s user 0.028s sys 0.051s
W20250623 01:58:41.408784 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 751552us
I20250623 01:58:41.297741 13954 rpcz_store.cc:276] 0623 01:58:39.934374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.187418 (+1253044us) service_pool.cc:225] Handling call
0623 01:58:41.297327 (+109909us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:41.342381 14793 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:41.345703 14515 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48412 (request call id 43) took 2523 ms (client timeout 2800 ms). Trace:
I20250623 01:58:41.347509 14769 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:41.348489 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:41.421775 13886 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:41.422359 13886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 975) took 1237 ms. Trace:
W20250623 01:58:41.419414 14515 rpcz_store.cc:269] 0623 01:58:38.821937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:38.822140 (+ 203us) service_pool.cc:225] Handling call
0623 01:58:41.345391 (+2523251us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b540054db54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:41.345587 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3451793792}
I20250623 01:58:41.351234 13955 rpcz_store.cc:276] 0623 01:58:39.870618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.186020 (+1315402us) service_pool.cc:225] Handling call
0623 01:58:41.187309 (+ 1289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:41.361850 13810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 433) took 2878 ms. Trace:
I20250623 01:58:41.423906 13810 rpcz_store.cc:276] 0623 01:58:38.483030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:38.483182 (+ 152us) service_pool.cc:225] Handling call
0623 01:58:38.568384 (+ 85202us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:38.652898 (+ 84514us) raft_consensus.cc:1532] Early marking committed up to index 3
0623 01:58:38.676808 (+ 23910us) log.cc:844] Serialized 14 byte log entry
0623 01:58:39.020623 (+343815us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:39.020694 (+ 71us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:39.020751 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:41.361517 (+2340766us) spinlock_profiling.cc:243] Waited 84.3 ms on lock 0x7b54001946d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:41.361583 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:41.361756 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40728,"raft.queue_time_us":13813,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":13457,"spinlock_wait_cycles":193846144,"thread_start_us":337141,"threads_started":1,"wal-append.queue_time_us":466639,"wal-append.run_cpu_time_us":6949,"wal-append.run_wall_time_us":1671204}
W20250623 01:58:41.362874 13962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 953) took 1542 ms (client timeout 1828 ms). Trace:
W20250623 01:58:41.426394 13962 rpcz_store.cc:269] 0623 01:58:39.819981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:39.820086 (+ 105us) service_pool.cc:225] Handling call
0623 01:58:41.362651 (+1542565us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b54000bc2d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:41.362778 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":60672256}
W20250623 01:58:41.367381 14907 log.cc:927] Time spent T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 2.395s user 0.000s sys 0.011s
I20250623 01:58:41.368851 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:41.370008 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:41.376149 13960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1011) took 2270 ms. Trace:
W20250623 01:58:41.376935 13779 connection.cc:232] RPC call timeout handler was delayed by -1.0326934649999657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20250623 01:58:41.377178 13776 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.029s sys 0.057s
W20250623 01:58:41.431918 13776 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.771s user 0.030s sys 0.057s
I20250623 01:58:41.086216 14942 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.169044 13957 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:40.705971 14858 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:41.080684 14879 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.185070 14938 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:40.977003 14948 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:40.978123 14941 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:41.093962 14770 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.172699 13879 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:40.979276 14952 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:41.171864 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 1.06359e+06us
I20250623 01:58:41.337911 13881 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:41.386016 14875 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:41.108144 14741 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.402216 13959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1012) took 2174 ms. Trace:
I20250623 01:58:41.429911 14916 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:41.081957 13808 rpcz_store.cc:269] 0623 01:58:39.063054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:39.073185 (+ 10131us) service_pool.cc:225] Handling call
0623 01:58:41.055807 (+1982622us) spinlock_profiling.cc:243] Waited 8.32 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:41.080011 (+ 24204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19129600}
I20250623 01:58:41.422690 13886 rpcz_store.cc:276] 0623 01:58:40.186005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.188609 (+ 2604us) service_pool.cc:225] Handling call
0623 01:58:41.348319 (+1159710us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:41.422295 (+ 73976us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31929088}
I20250623 01:58:41.339326 14897 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:41.105193 14917 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.431674 13960 rpcz_store.cc:276] 0623 01:58:39.106282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:39.106591 (+ 309us) service_pool.cc:225] Handling call
0623 01:58:41.375318 (+2268727us) spinlock_profiling.cc:243] Waited 968 us on lock 0x7b5800016e98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:58:41.375556 (+ 238us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:41.375777 (+ 221us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:41.375813 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:41.375868 (+ 55us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:41.375906 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:41.375977 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:41.376075 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2226560}
I20250623 01:58:41.418869 14785 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:58:41.380559 14512 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48402 (request call id 81) took 2252 ms (client timeout 1534 ms). Trace:
I20250623 01:58:41.380425 13953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 974) took 1393 ms. Trace:
I20250623 01:58:41.380290 13887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 971) took 1162 ms. Trace:
I20250623 01:58:41.109326 14821 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:41.544543 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 91739.5us
I20250623 01:58:41.555689 14876 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:41.564972 13957 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 981) took 1613 ms (client timeout 1953 ms). Trace:
I20250623 01:58:41.609330 14966 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:41.627240 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:41.665520 13954 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:41.666026 13954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 1016) took 1555 ms (client timeout 1967 ms). Trace:
I20250623 01:58:41.608631 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:41.609196 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:41.093739 14935 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.631743 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:41.609097 14946 log.cc:927] Time spent T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.557s user 0.000s sys 0.001s
I20250623 01:58:41.552584 14853 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:41.636461 13883 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:41.638773 13959 rpcz_store.cc:276] 0623 01:58:39.228042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:39.228284 (+ 242us) service_pool.cc:225] Handling call
0623 01:58:41.353159 (+2124875us) spinlock_profiling.cc:243] Waited 890 us on lock 0x7b5800016e98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:58:41.401617 (+ 48458us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:41.401832 (+ 215us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:41.401869 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:41.401924 (+ 55us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:41.401964 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:41.402030 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:41.402152 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2047232}
I20250623 01:58:41.418746 14795 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:41.633211 13804 raft_consensus.cc:2391] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:41.565106 14948 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:41.643360 14974 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:41.634725 13884 raft_consensus.cc:2391] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:41.616883 14791 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:41.555374 14954 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:41.611625 14794 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:41.649139 14875 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.653115 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:41.103179 14944 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.141831 14930 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:41.657402 13927 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.008s sys 0.032s
I20250623 01:58:41.658809 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 418) took 3402 ms. Trace:
I20250623 01:58:41.659489 13958 raft_consensus.cc:2391] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:41.429585 14793 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.450752 13806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41526 (request call id 42) took 1282 ms. Trace:
I20250623 01:58:41.636636 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41526 (request call id 36) took 3063 ms. Trace:
I20250623 01:58:41.667426 13962 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:41.434924 14769 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:41.451171 14958 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:41.647305 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:41.633154 14938 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.644611 14941 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.438657 14945 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:41.645236 14952 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:41.666376 13954 rpcz_store.cc:269] 0623 01:58:40.110276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.626920 (+1516644us) service_pool.cc:225] Handling call
0623 01:58:41.665972 (+ 39052us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:42.245728 13927 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.135s user 0.027s sys 0.045s
I20250623 01:58:41.650977 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 983) took 1363 ms. Trace:
I20250623 01:58:41.849967 14955 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:41.701292 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:58:41.734043 13956 consensus_meta.cc:287] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 1.039s user 0.009s sys 0.000s
I20250623 01:58:41.649389 14785 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:42.320065 13959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 978) took 2369 ms (client timeout 2245 ms). Trace:
I20250623 01:58:42.320379 13956 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:42.256273 13955 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
W20250623 01:58:42.320890 13956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 985) took 2245 ms (client timeout 1845 ms). Trace:
W20250623 01:58:42.321053 13955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 975) took 2372 ms (client timeout 2114 ms). Trace:
W20250623 01:58:42.321228 13956 rpcz_store.cc:269] 0623 01:58:40.076029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.687373 (+611344us) service_pool.cc:225] Handling call
0623 01:58:42.320828 (+1633455us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:42.321523 13955 rpcz_store.cc:269] 0623 01:58:39.949253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.631429 (+1682176us) service_pool.cc:225] Handling call
0623 01:58:42.320997 (+689568us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.257480 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:42.063303 14943 log.cc:927] Time spent T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.577s user 0.002s sys 0.000s
W20250623 01:58:42.020284 14882 consensus_meta.cc:287] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 2.184s user 0.014s sys 0.000s
I20250623 01:58:42.123050 14821 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:41.652014 13953 rpcz_store.cc:276] 0623 01:58:39.935793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.268757 (+332964us) service_pool.cc:225] Handling call
0623 01:58:41.328770 (+1060013us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.253185 14836 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:41.690788 14866 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:41.703446 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 58326.2us
I20250623 01:58:41.905828 14927 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:41.913190 14914 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:41.784137 14767 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:42.240744 14923 consensus_meta.cc:287] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 2.355s user 0.011s sys 0.001s
W20250623 01:58:42.219980 14911 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.000s sys 0.017s
I20250623 01:58:41.843318 14876 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:42.333352 14775 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.013s sys 0.011s
I20250623 01:58:41.740343 14924 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:41.892508 14968 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:41.660866 14932 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.154561 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:41.668084 14846 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:41.945802 14962 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
W20250623 01:58:42.345779 14775 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s user 0.013s sys 0.011s
I20250623 01:58:41.939254 14879 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:42.268443 14931 consensus_meta.cc:287] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 2.071s user 0.005s sys 0.012s
I20250623 01:58:42.255182 14942 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:42.349453 14911 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.989s user 0.001s sys 0.018s
I20250623 01:58:41.910372 14764 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:41.939817 14910 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:41.647426 14916 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:42.254768 14770 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.043418 14831 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:42.322118 14741 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:41.784113 13778 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.011s sys 0.024s
W20250623 01:58:42.410296 13776 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.011s sys 0.006s
W20250623 01:58:42.410892 13884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 973) took 2233 ms (client timeout 1755 ms). Trace:
I20250623 01:58:42.258651 13886 raft_consensus.cc:2391] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:42.362531 14916 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:42.411684 13804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 1002) took 3350 ms (client timeout 1564 ms). Trace:
W20250623 01:58:42.257995 13928 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.919s user 0.017s sys 0.041s
W20250623 01:58:42.435125 13928 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.096s user 0.017s sys 0.041s
W20250623 01:58:42.435292 13886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 978) took 2248 ms (client timeout 2217 ms). Trace:
I20250623 01:58:42.436024 14935 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:42.180145 13885 consensus_meta.cc:287] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.949s user 0.005s sys 0.004s
I20250623 01:58:42.437245 14741 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:42.437776 13885 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:42.438575 14941 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.438300 14741 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:42.412644 13777 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.002s sys 0.008s
W20250623 01:58:41.877712 13852 connection.cc:232] RPC call timeout handler was delayed by -1.9597117120000576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20250623 01:58:42.439699 13777 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.595s user 0.002s sys 0.008s
W20250623 01:58:42.323940 13954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 977) took 2374 ms (client timeout 2217 ms). Trace:
W20250623 01:58:42.327526 13955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 983) took 2279 ms (client timeout 2172 ms). Trace:
W20250623 01:58:42.327961 13956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 979) took 2377 ms (client timeout 2246 ms). Trace:
I20250623 01:58:42.251585 13882 rpcz_store.cc:276] 0623 01:58:38.248285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:38.248483 (+ 198us) service_pool.cc:225] Handling call
0623 01:58:38.248877 (+ 394us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:38.249276 (+ 399us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:38.249333 (+ 57us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:38.249850 (+ 517us) log.cc:844] Serialized 26 byte log entry
0623 01:58:38.795281 (+545431us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:38.795334 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:38.795407 (+ 73us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:41.366530 (+2571123us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:58:41.644719 (+278189us) raft_consensus.cc:1683] finished
0623 01:58:41.644742 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:41.650613 (+ 5871us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4338,"raft.queue_time_us":5083,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":10265,"spinlock_wait_cycles":4534016,"thread_start_us":532319,"threads_started":1,"wal-append.queue_time_us":649814}
W20250623 01:58:42.320535 13959 rpcz_store.cc:269] 0623 01:58:39.950218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.316777 (+2366559us) service_pool.cc:212] Skipping call since client already timed out
0623 01:58:42.320008 (+ 3231us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250623 01:58:42.325448 14508 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:41.648474 14512 rpcz_store.cc:269] 0623 01:58:39.076179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:39.079778 (+ 3599us) service_pool.cc:225] Handling call
0623 01:58:40.197578 (+1117800us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b58001e2a98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:41.328404 (+1130826us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13988224}
I20250623 01:58:42.449223 14846 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:42.264891 14791 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:42.450222 14846 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:42.450822 13882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 979) took 2263 ms (client timeout 2245 ms). Trace:
I20250623 01:58:42.451372 14741 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:42.451270 13882 rpcz_store.cc:269] 0623 01:58:40.187252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.450396 (+2263144us) service_pool.cc:212] Skipping call since client already timed out
0623 01:58:42.450642 (+ 246us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250623 01:58:42.452251 13882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 980) took 2264 ms (client timeout 2246 ms). Trace:
I20250623 01:58:42.259256 14794 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:42.238512 14849 consensus_meta.cc:287] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Time spent flushing consensus metadata: real 1.512s user 0.011s sys 0.000s
I20250623 01:58:42.123508 14858 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:42.348212 14876 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.331321 14882 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:42.453938 14849 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:41.652617 14897 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.356592 14917 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.363526 14769 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:42.357357 14931 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:42.419382 14944 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.370996 14938 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.349851 14914 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.372659 14945 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.331882 14955 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.326316 14954 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.273617 14958 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:42.347563 14923 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:42.482895 13961 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:42.365159 14962 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:42.073088 14964 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:42.163908 14965 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:41.691071 14966 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.508913 13962 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:42.329540 14967 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:42.241099 14970 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:42.265022 14974 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.346220 14767 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:42.424589 14509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48402 (request call id 84) took 1027 ms. Trace:
I20250623 01:58:42.424686 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.343528 14513 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:42.426406 13880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 27) took 3268 ms. Trace:
W20250623 01:58:42.220185 13779 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.000s sys 0.019s
I20250623 01:58:42.272971 13881 rpcz_store.cc:276] 0623 01:58:40.278901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.708211 (+429310us) service_pool.cc:225] Handling call
0623 01:58:41.642171 (+933960us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:41.650755 13887 rpcz_store.cc:276] 0623 01:58:40.167457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.182105 (+ 14648us) service_pool.cc:225] Handling call
0623 01:58:41.296698 (+1114593us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:41.664726 13957 rpcz_store.cc:269] 0623 01:58:39.951090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.318849 (+367759us) service_pool.cc:225] Handling call
0623 01:58:41.564909 (+1246060us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.418272 14930 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:42.430531 13884 rpcz_store.cc:269] 0623 01:58:40.177256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.180730 (+ 3474us) service_pool.cc:225] Handling call
0623 01:58:41.369827 (+1189097us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:42.410521 (+1040694us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6625024}
I20250623 01:58:42.419724 14926 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:42.431959 13776 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.769s user 0.012s sys 0.006s
W20250623 01:58:42.435938 13886 rpcz_store.cc:269] 0623 01:58:40.186964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.652830 (+1465866us) service_pool.cc:225] Handling call
0623 01:58:42.434988 (+782158us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:42.439499 13885 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 402) took 5123 ms (client timeout 1894 ms). Trace:
I20250623 01:58:42.530205 14926 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:42.439980 13852 connection.cc:232] RPC call timeout handler was delayed by -1.778342551000037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1502s for a response
W20250623 01:58:42.413146 13926 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.006s sys 0.013s
W20250623 01:58:42.412420 13853 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.000s sys 0.016s
I20250623 01:58:42.441133 14886 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:42.441358 13956 rpcz_store.cc:269] 0623 01:58:39.950510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.327814 (+2377304us) service_pool.cc:212] Skipping call since client already timed out
0623 01:58:42.327911 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250623 01:58:42.441198 13955 rpcz_store.cc:269] 0623 01:58:40.043079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.322199 (+2279120us) service_pool.cc:212] Skipping call since client already timed out
0623 01:58:42.322303 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250623 01:58:42.413643 13806 rpcz_store.cc:276] 0623 01:58:40.153534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.153690 (+ 156us) service_pool.cc:225] Handling call
0623 01:58:40.154064 (+ 374us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:40.154456 (+ 392us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:40.154497 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:40.182016 (+ 27519us) log.cc:844] Serialized 26 byte log entry
0623 01:58:40.347132 (+165116us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:58:40.347174 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:40.347239 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:41.435322 (+1088083us) raft_consensus.cc:1683] finished
0623 01:58:41.435340 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:41.435447 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3610,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":7052,"thread_start_us":158873,"threads_started":1,"wal-append.queue_time_us":610721}
I20250623 01:58:42.444269 14987 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:42.444461 14978 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:42.452790 14846 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:42.452699 13882 rpcz_store.cc:269] 0623 01:58:40.187557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.452064 (+2264507us) service_pool.cc:212] Skipping call since client already timed out
0623 01:58:42.452178 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250623 01:58:42.456317 13778 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.270s user 0.016s sys 0.045s
I20250623 01:58:42.457405 14876 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.448961 14875 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.419490 13811 rpcz_store.cc:276] 0623 01:58:38.571116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:38.574232 (+ 3116us) service_pool.cc:225] Handling call
0623 01:58:38.579243 (+ 5011us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:39.042639 (+463396us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:39.053363 (+ 10724us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:39.143651 (+ 90288us) log.cc:844] Serialized 26 byte log entry
0623 01:58:39.446796 (+303145us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:58:39.459179 (+ 12383us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:39.488516 (+ 29337us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:41.565912 (+2077396us) raft_consensus.cc:1683] finished
0623 01:58:41.587776 (+ 21864us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:41.633981 (+ 46205us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":69377,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":632483,"threads_started":2,"wal-append.queue_time_us":897793}
I20250623 01:58:42.437729 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.461992 14897 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.452615 14791 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:42.463670 14929 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:42.441025 13954 rpcz_store.cc:269] 0623 01:58:39.949931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.319109 (+2369178us) service_pool.cc:212] Skipping call since client already timed out
0623 01:58:42.323859 (+ 4750us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250623 01:58:42.466328 14941 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:42.458097 14916 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.457141 14948 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:42.444334 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 539151us
W20250623 01:58:42.439190 13804 rpcz_store.cc:269] 0623 01:58:39.061324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:39.072857 (+ 11533us) service_pool.cc:225] Handling call
0623 01:58:41.368671 (+2295814us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b031ec 00007f41e0b02dab 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:42.411342 (+1042671us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7533440}
I20250623 01:58:42.473807 14849 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.474436 14954 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.473809 14847 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20250623 01:58:42.552740 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:42.554683 14941 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:42.474959 14958 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:42.475188 14514 raft_consensus.cc:1238] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:42.360749 14764 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.477330 14793 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.480057 14882 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:42.480248 14955 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.481464 14952 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.484522 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.565346 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:42.566344 13879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 970) took 2400 ms (client timeout 2115 ms). Trace:
I20250623 01:58:42.489405 14914 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.492383 14962 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:42.491102 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:42.363765 14831 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.487984 14964 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.494628 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.484773 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:42.498116 13883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 987) took 1805 ms (client timeout 1534 ms). Trace:
I20250623 01:58:42.499191 14794 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.502036 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 171) took 5500 ms. Trace:
I20250623 01:58:42.502358 14510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48412 (request call id 47) took 1853 ms. Trace:
I20250623 01:58:42.578821 13809 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:42.502853 14785 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:42.508330 13958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 967) took 2609 ms (client timeout 2060 ms). Trace:
I20250623 01:58:42.356094 14968 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:42.510316 13962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 962) took 2613 ms (client timeout 2196 ms). Trace:
I20250623 01:58:42.512696 13960 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:42.512984 14966 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:42.582136 13962 rpcz_store.cc:269] 0623 01:58:39.897384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.608915 (+1711531us) service_pool.cc:225] Handling call
0623 01:58:42.510207 (+901292us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:42.518052 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 62813.1us
I20250623 01:58:42.584476 13888 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:42.508021 14795 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.585988 14831 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.586979 14795 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.587128 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:42.588351 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:42.588521 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.589921 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:42.520505 14970 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.522401 14965 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.520284 14967 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.524233 13880 rpcz_store.cc:276] 0623 01:58:39.158281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:39.159673 (+ 1392us) service_pool.cc:225] Handling call
0623 01:58:39.213664 (+ 53991us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:39.213976 (+ 312us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:39.214019 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:39.214557 (+ 538us) log.cc:844] Serialized 26 byte log entry
0623 01:58:39.840534 (+625977us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b18002b10e0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b22bc5 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e36dd04b 00007f41e36e2b53 00007f41e36e2f01 00007f41e3a71ccf
0623 01:58:40.334760 (+494226us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:58:40.334817 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:40.334909 (+ 92us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:42.425661 (+2090752us) raft_consensus.cc:1683] finished
0623 01:58:42.425724 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:42.426136 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12522,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":35435,"spinlock_wait_cycles":7242496,"thread_start_us":493991,"threads_started":1,"wal-append.queue_time_us":1745246}
I20250623 01:58:42.529350 14974 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:42.531219 13885 rpcz_store.cc:269] 0623 01:58:37.316130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:37.348485 (+ 32355us) service_pool.cc:225] Handling call
0623 01:58:42.439183 (+5090698us) spinlock_profiling.cc:243] Waited 3.83 s on lock 0x7b54001a5254. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:42.439387 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8810691456}
I20250623 01:58:42.533390 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:42.533977 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 871462us
W20250623 01:58:42.534508 13779 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.096s user 0.000s sys 0.020s
I20250623 01:58:42.535274 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:42.532950 13953 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:42.536445 13926 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.898s user 0.006s sys 0.013s
I20250623 01:58:42.055980 14922 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:42.538170 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.539367 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.531646 14767 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:42.531728 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.543138 13853 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.000s sys 0.016s
I20250623 01:58:42.525811 14923 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.541745 14987 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.527490 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:42.617177 13881 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:42.617354 13953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 1015) took 2499 ms (client timeout 2696 ms). Trace:
W20250623 01:58:42.617904 13953 rpcz_store.cc:269] 0623 01:58:40.109898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.423792 (+2313894us) service_pool.cc:225] Handling call
0623 01:58:42.608991 (+185199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.618860 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.523417 14509 rpcz_store.cc:276] 0623 01:58:41.396581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.433859 (+ 37278us) service_pool.cc:225] Handling call
0623 01:58:42.123760 (+689901us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b58001e2a98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:42.424223 (+300463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38715264}
I20250623 01:58:42.621434 14831 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:42.621553 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.544716 14978 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.551043 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.550580 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.547883 14886 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.566824 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:42.573218 13883 rpcz_store.cc:269] 0623 01:58:40.692296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.109106 (+416810us) service_pool.cc:225] Handling call
0623 01:58:42.497732 (+1388626us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b540011ec54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:42.497937 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1060153088}
I20250623 01:58:42.567688 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.578994 13813 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:42.567058 14941 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:42.580227 13958 rpcz_store.cc:269] 0623 01:58:39.898820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.691683 (+792863us) service_pool.cc:225] Handling call
0623 01:58:42.507810 (+1816127us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b540027bd54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:42.508100 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2192424832}
I20250623 01:58:42.610790 13809 raft_consensus.cc:1238] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:42.610760 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.614339 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 1.17841e+06us
W20250623 01:58:42.566885 13879 rpcz_store.cc:269] 0623 01:58:40.165712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.182074 (+ 16362us) service_pool.cc:225] Handling call
0623 01:58:42.565949 (+2383875us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b54001a4d54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:42.566126 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57462144}
I20250623 01:58:42.614815 13887 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:42.615972 13961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 968) took 2682 ms (client timeout 1879 ms). Trace:
I20250623 01:58:42.616349 13886 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
W20250623 01:58:42.632851 13961 rpcz_store.cc:269] 0623 01:58:39.934008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.159525 (+225517us) service_pool.cc:225] Handling call
0623 01:58:42.615707 (+2456182us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b540027a954. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:42.615881 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5281340800}
I20250623 01:58:42.616008 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:42.638434 13886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 992) took 1586 ms (client timeout 1637 ms). Trace:
W20250623 01:58:42.638902 13886 rpcz_store.cc:269] 0623 01:58:41.048413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.531371 (+1482958us) service_pool.cc:225] Handling call
0623 01:58:42.634270 (+102899us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.640686 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.642201 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.642864 13962 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:42.643819 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.647444 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:42.645157 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.648797 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.651125 13884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 990) took 1726 ms (client timeout 1784 ms). Trace:
W20250623 01:58:42.651661 13884 rpcz_store.cc:269] 0623 01:58:40.925172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.537689 (+1612517us) service_pool.cc:225] Handling call
0623 01:58:42.651051 (+113362us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.622301 14804 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:42.652961 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.654867 13880 raft_consensus.cc:2391] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:42.630712 13953 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:42.622150 14831 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.628350 14966 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:42.660748 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.661900 13885 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 994) took 1609 ms (client timeout 1794 ms). Trace:
W20250623 01:58:42.666596 13885 rpcz_store.cc:269] 0623 01:58:41.049156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.598610 (+1549454us) service_pool.cc:225] Handling call
0623 01:58:42.658462 (+ 59852us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:42.662060 13880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 998) took 1609 ms (client timeout 1949 ms). Trace:
W20250623 01:58:42.667286 13880 rpcz_store.cc:269] 0623 01:58:41.050413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.613038 (+1562625us) service_pool.cc:225] Handling call
0623 01:58:42.659802 (+ 46764us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.665913 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.668560 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.647207 13881 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:42.669142 13882 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:42.669694 13955 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:42.671610 13959 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.673111 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.674243 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:42.674175 13882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 986) took 2025 ms (client timeout 2172 ms). Trace:
W20250623 01:58:42.678803 13882 rpcz_store.cc:269] 0623 01:58:40.648596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.543156 (+1894560us) service_pool.cc:225] Handling call
0623 01:58:42.674099 (+130943us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.579277 14510 rpcz_store.cc:276] 0623 01:58:40.648807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.652029 (+ 3222us) service_pool.cc:225] Handling call
0623 01:58:40.726215 (+ 74186us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:41.728316 (+1002101us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:41.728367 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:41.728435 (+ 68us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:41.728487 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:42.501870 (+773383us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b540056dbd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:42.501944 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:42.502161 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2263603456}
I20250623 01:58:42.670112 14785 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.679700 13956 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
W20250623 01:58:42.684252 13886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 991) took 1657 ms (client timeout 2084 ms). Trace:
W20250623 01:58:42.684546 13886 rpcz_store.cc:269] 0623 01:58:41.026924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.645394 (+1618470us) service_pool.cc:225] Handling call
0623 01:58:42.684209 (+ 38815us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:42.686460 13884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 999) took 1633 ms (client timeout 2098 ms). Trace:
W20250623 01:58:42.686990 13884 rpcz_store.cc:269] 0623 01:58:41.050710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.652622 (+1601912us) service_pool.cc:225] Handling call
0623 01:58:42.681828 (+ 29206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.688220 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.689596 14966 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.690404 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:42.691864 14966 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.694530 14831 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.695473 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.696033 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.696790 13808 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.696950 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:42.697688 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.698367 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:42.698777 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:42.681551 13954 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:42.702291 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.702654 14987 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.663193 13957 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.708596 13958 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:42.713361 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.577919 13710 rpcz_store.cc:276] 0623 01:58:37.001340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:37.004153 (+ 2813us) service_pool.cc:225] Handling call
0623 01:58:42.501682 (+5497529us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:37.057287 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:37.057843 (+ 556us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:37.057872 (+ 29us) write_op.cc:435] Acquired schema lock
0623 01:58:37.057888 (+ 16us) tablet.cc:660] Decoding operations
0623 01:58:37.058253 (+ 365us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:37.058305 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:37.058333 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0623 01:58:37.058654 (+ 321us) tablet.cc:699] Row locks acquired
0623 01:58:37.058670 (+ 16us) write_op.cc:260] PREPARE: finished
0623 01:58:37.058782 (+ 112us) write_op.cc:270] Start()
0623 01:58:37.058877 (+ 95us) write_op.cc:276] Timestamp: P: 1750643917058744 usec, L: 0
0623 01:58:37.058900 (+ 23us) op_driver.cc:348] REPLICATION: starting
0623 01:58:37.164528 (+105628us) log.cc:844] Serialized 972 byte log entry
0623 01:58:37.944891 (+780363us) op_driver.cc:464] REPLICATION: finished
0623 01:58:38.587807 (+642916us) write_op.cc:301] APPLY: starting
0623 01:58:38.594157 (+ 6350us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:38.604754 (+ 10597us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:38.657008 (+ 52254us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:38.687799 (+ 30791us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:38.695456 (+ 7657us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 01:58:38.695507 (+ 51us) write_op.cc:312] APPLY: finished
0623 01:58:38.695684 (+ 177us) log.cc:844] Serialized 28 byte log entry
0623 01:58:38.695829 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:38.695922 (+ 93us) write_op.cc:454] Released schema lock
0623 01:58:38.695977 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:38.696318 (+ 341us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:38.696343 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":46972,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":611523,"apply.run_cpu_time_us":2343,"apply.run_wall_time_us":114846,"num_ops":2,"prepare.queue_time_us":977,"prepare.run_cpu_time_us":14539,"prepare.run_wall_time_us":204063,"replication_time_us":872591,"thread_start_us":258783,"threads_started":1}]]}
W20250623 01:58:42.681854 13881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 988) took 1909 ms (client timeout 1845 ms). Trace:
I20250623 01:58:42.689145 13884 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:42.717649 13961 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:42.716141 13883 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
W20250623 01:58:42.719478 13883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 995) took 1670 ms (client timeout 1998 ms). Trace:
W20250623 01:58:42.719882 13883 rpcz_store.cc:269] 0623 01:58:41.049492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.637087 (+1587595us) service_pool.cc:225] Handling call
0623 01:58:42.719415 (+ 82328us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.717864 13885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1000) took 1158 ms. Trace:
I20250623 01:58:42.720547 13885 rpcz_store.cc:276] 0623 01:58:41.559109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.672054 (+1112945us) service_pool.cc:225] Handling call
0623 01:58:42.717821 (+ 45767us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.699986 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.695511 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.699932 14970 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.727922 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:42.727385 13813 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.715317 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.718461 14965 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.700299 14804 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.726639 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:42.722687 14849 consensus_queue.cc:1046] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.014s
I20250623 01:58:42.794111 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.790187 13954 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250623 01:58:42.807241 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:42.735364 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.797705 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 43) took 2911 ms. Trace:
I20250623 01:58:42.741789 13886 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:42.808830 13955 rpcz_store.cc:276] 0623 01:58:39.886143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.683157 (+2797014us) service_pool.cc:225] Handling call
0623 01:58:42.724362 (+ 41205us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:42.797344 (+ 72982us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:42.797369 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:42.797414 (+ 45us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:58:42.797444 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:42.797520 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:42.797632 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:42.716715 13881 rpcz_store.cc:269] 0623 01:58:40.767764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.526956 (+1759192us) service_pool.cc:225] Handling call
0623 01:58:42.676555 (+149599us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.807459 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:42.805683 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 116376us
I20250623 01:58:42.793507 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37408 (request call id 353) took 2606 ms. Trace:
W20250623 01:58:42.806452 13960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 961) took 2909 ms (client timeout 2235 ms). Trace:
I20250623 01:58:42.806097 14514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48402 (request call id 82) took 3524 ms. Trace:
I20250623 01:58:42.817019 14514 rpcz_store.cc:276] 0623 01:58:39.282155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:39.282392 (+ 237us) service_pool.cc:225] Handling call
0623 01:58:39.826701 (+544309us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:42.558078 (+2731377us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:42.805750 (+247672us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b54004902d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:42.806006 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6086549248}
I20250623 01:58:42.818150 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.819124 13811 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:42.818274 13812 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:42.775987 14994 log.cc:927] Time spent T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250623 01:58:42.794875 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:42.778051 14997 log.cc:927] Time spent T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20250623 01:58:42.792245 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:42.790853 13925 leader_election.cc:336] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 1.529s (SENT)
I20250623 01:58:42.813868 13961 rpcz_store.cc:276] 0623 01:58:40.187574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.722376 (+2534802us) service_pool.cc:225] Handling call
0623 01:58:42.793405 (+ 71029us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.736778 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.814590 13960 rpcz_store.cc:269] 0623 01:58:39.897045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.654005 (+1756960us) service_pool.cc:225] Handling call
0623 01:58:42.806274 (+1152269us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54000a3f54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:42.806395 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430453504}
I20250623 01:58:42.793721 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.742008 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.793723 13804 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.819473 13779 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:42.743992 13879 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:42.836645 13956 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:42.828955 13962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1017) took 2718 ms. Trace:
I20250623 01:58:42.831035 13808 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.831712 13807 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.825217 13810 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:42.845764 13884 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:58:42.823921 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 95173.6us
I20250623 01:58:42.852303 13885 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
W20250623 01:58:42.853168 13925 leader_election.cc:336] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 1.894s (SENT)
I20250623 01:58:42.853670 13925 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:42.835839 13806 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.828826 14927 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:42.856479 13953 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.856434 14927 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:42.858618 14927 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.785660 14967 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.859449 14949 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:42.860621 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.861392 13805 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:42.863413 14478 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:42.864208 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.864538 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.838094 13962 rpcz_store.cc:276] 0623 01:58:40.110567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.704375 (+2593808us) service_pool.cc:225] Handling call
0623 01:58:42.730518 (+ 26143us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:42.821943 (+ 91425us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:42.821966 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:42.824458 (+ 2492us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:42.828708 (+ 4250us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:42.828779 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:42.828884 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.737046 14764 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.840874 14886 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.865696 13810 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
W20250623 01:58:42.866554 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 47141.9us
I20250623 01:58:42.734380 14978 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:42.742197 14926 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.861204 14804 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:42.865517 13879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 31) took 1247 ms (client timeout 1617 ms). Trace:
I20250623 01:58:42.897634 14979 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:42.898823 14968 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:42.899628 13879 rpcz_store.cc:269] 0623 01:58:41.620877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.662987 (+1042110us) service_pool.cc:225] Handling call
0623 01:58:42.855244 (+192257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:42.901087 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.901895 13879 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:42.902415 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:42.903048 13954 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:42.903215 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.904129 13879 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:42.904476 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.866427 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.909021 13883 raft_consensus.cc:2366] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:42.910295 14478 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:42.911186 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:42.892364 14967 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:42.914824 13883 raft_consensus.cc:2374] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2. Re-sending same reply.
I20250623 01:58:42.916301 13779 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:42.917375 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.916688 14821 raft_consensus.cc:2802] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:42.921944 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.922506 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.922689 13808 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:42.922746 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.923585 13885 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:42.925122 13882 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:42.926013 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.927495 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 74401.4us
I20250623 01:58:42.924388 13881 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:42.931739 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.935005 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.935703 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.928140 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.941673 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.944396 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.945286 13804 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:42.947351 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.948740 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.939741 14775 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:42.938319 14968 raft_consensus.cc:2802] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:42.951670 14775 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:42.952056 14968 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:42.952165 14775 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:42.952582 14968 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:42.956364 14964 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:42.936633 13887 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:42.949959 13885 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:42.961360 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.963554 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.971333 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.949175 13879 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
W20250623 01:58:42.972823 13926 connection.cc:232] RPC call timeout handler was delayed by -1.1782921230000056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
I20250623 01:58:42.975489 13885 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
W20250623 01:58:42.976980 13926 connection.cc:232] RPC call timeout handler was delayed by -1.074792122999952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
W20250623 01:58:42.977576 13926 connection.cc:232] RPC call timeout handler was delayed by -1.0675921229999403s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
I20250623 01:58:42.977882 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.962803 13925 leader_election.cc:336] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 1.644s (SENT)
I20250623 01:58:42.982636 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.986222 13882 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:42.987322 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.969889 13887 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 993) took 1912 ms (client timeout 1648 ms). Trace:
W20250623 01:58:42.984009 13888 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 982) took 2782 ms (client timeout 1668 ms). Trace:
W20250623 01:58:42.990877 13888 rpcz_store.cc:269] 0623 01:58:40.202028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:40.657058 (+455030us) service_pool.cc:225] Handling call
0623 01:58:42.977373 (+2320315us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54001a4354. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:42.983920 (+ 6547us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3444663552}
I20250623 01:58:42.991719 13885 raft_consensus.cc:2391] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
W20250623 01:58:42.988240 13925 leader_election.cc:336] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 1.652s (SENT)
I20250623 01:58:42.995837 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:42.970340 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:42.997874 13883 raft_consensus.cc:2408] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:42.998971 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.001729 13806 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:42.988129 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:42.996526 13882 raft_consensus.cc:2374] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3. Re-sending same reply.
I20250623 01:58:43.000339 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.011008 13885 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:42.999258 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.000149 14821 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:58:43.016279 15002 log.cc:927] Time spent T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20250623 01:58:43.016307 14821 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.013819 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.020675 13886 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:43.021569 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.021834 14478 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:43.019892 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:42.990818 13887 rpcz_store.cc:269] 0623 01:58:41.048843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.534914 (+1486071us) service_pool.cc:225] Handling call
0623 01:58:42.959607 (+424693us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:43.024735 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:43.004282 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 73984.5us
I20250623 01:58:43.026624 13880 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:43.029618 13880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1002) took 1469 ms. Trace:
I20250623 01:58:43.029966 13880 rpcz_store.cc:276] 0623 01:58:41.559755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.667737 (+1107982us) service_pool.cc:225] Handling call
0623 01:58:43.019754 (+352017us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:43.029565 (+ 9811us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14925184}
I20250623 01:58:43.015738 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.031769 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:43.013644 13881 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:43.033754 13888 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
W20250623 01:58:43.037942 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 61398.3us
I20250623 01:58:43.033284 13884 raft_consensus.cc:2366] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.043603 14745 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:43.047698 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:43.047883 15000 log.cc:927] Time spent T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20250623 01:58:43.048288 13954 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:43.048230 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.051667 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.042032 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.066550 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.067108 14849 raft_consensus.cc:3053] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:43.067507 14849 raft_consensus.cc:738] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:43.067658 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.068378 14849 consensus_queue.cc:260] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:43.071188 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.074515 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.074762 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.075146 13955 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:43.075366 13960 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:43.075719 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.076006 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.076248 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.076373 13812 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:43.045415 14745 raft_consensus.cc:695] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:43.077944 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.073539 13954 raft_consensus.cc:2391] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:43.077144 13960 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:43.086508 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.088286 13812 raft_consensus.cc:1238] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:43.080575 13879 raft_consensus.cc:2391] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:43.085265 13809 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
W20250623 01:58:43.093493 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:43.102617 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.093158 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.104465 13960 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:43.088614 14849 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.109463 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:43.113886 13812 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.113919 13779 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:43.114894 13954 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:43.111685 13776 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:43.118013 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.118189 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.118719 13809 raft_consensus.cc:2391] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:43.091688 13887 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:43.118932 13879 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:43.097102 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.127849 13959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 44) took 3036 ms. Trace:
I20250623 01:58:43.131888 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.134354 14979 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:43.136260 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.117573 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.132823 13959 rpcz_store.cc:276] 0623 01:58:40.091409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.682020 (+2590611us) service_pool.cc:225] Handling call
0623 01:58:42.724140 (+ 42120us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:42.734965 (+ 10825us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:42.734983 (+ 18us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:42.780431 (+ 45448us) log.cc:844] Serialized 26 byte log entry
0623 01:58:42.891155 (+110724us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:58:42.891208 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:42.891303 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:43.127610 (+236307us) raft_consensus.cc:1683] finished
0623 01:58:43.127638 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:43.127785 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":93133,"threads_started":1,"wal-append.queue_time_us":164284}
I20250623 01:58:43.160136 13776 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:43.125288 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.162084 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:43.169801 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.163333 13960 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:43.131165 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:43.175729 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 66667.1us
I20250623 01:58:43.177449 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:43.178027 13887 raft_consensus.cc:2374] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3. Re-sending same reply.
W20250623 01:58:43.178978 13852 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:58122 dropped due to backpressure: service queue is full with 50 items
I20250623 01:58:43.180214 13813 raft_consensus.cc:2391] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
W20250623 01:58:43.181758 15005 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20250623 01:58:43.165802 14969 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:43.185364 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.175442 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:43.185635 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.185174 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.178910 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.190981 13954 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.192130 13779 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:43.192263 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.192130 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.197252 13961 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:43.203703 13959 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:43.203855 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:43.204780 13805 raft_consensus.cc:2391] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:43.204854 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.196847 13957 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:43.205752 14979 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.206374 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.172191 14745 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:43.209406 13854 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:43.213143 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 53097.2us
I20250623 01:58:43.215368 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.216651 14882 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:43.194762 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:43.217835 13879 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:43.190675 14969 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:43.218425 14969 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.221911 13776 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.179289 13852 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:58122 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 456)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1039)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1040)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1008)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1028)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1013)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1023)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1024)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1009)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 33)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1041)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1042)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1032)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1043)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1044)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1033)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1034)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1035)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 459)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 30)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 985)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 989)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 996)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 997)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1001)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 32)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 423)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 425)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1010)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 446)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1030)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1031)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 455)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 37)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1036)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 460)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 461)
I20250623 01:58:43.224100 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.225083 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.198648 13953 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.225696 13960 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:43.230048 13776 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:43.231416 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.234047 13888 raft_consensus.cc:2391] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:43.238703 13958 raft_consensus.cc:2391] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:43.241623 13854 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.240193 14882 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:43.250808 13854 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.255508 13776 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:43.249666 14926 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.286180 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.286458 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.287014 13886 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:43.288123 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.289940 14513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48412 (request call id 48) took 1847 ms. Trace:
I20250623 01:58:43.265900 14970 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:43.296073 14964 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:43.290722 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.298009 13779 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:43.304139 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.273705 14926 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.290434 14513 rpcz_store.cc:276] 0623 01:58:41.381430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:41.396401 (+ 14971us) service_pool.cc:225] Handling call
0623 01:58:41.544448 (+148047us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:42.335968 (+791520us) raft_consensus.cc:1532] Early marking committed up to index 3
0623 01:58:42.726376 (+390408us) log.cc:844] Serialized 14 byte log entry
0623 01:58:42.855150 (+128774us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:42.887891 (+ 32741us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:42.894848 (+ 6957us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.211455 (+316607us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54004cd454. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:43.214994 (+ 3539us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:43.226685 (+ 11691us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11535,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"spinlock_wait_cycles":1800997760,"thread_start_us":95430,"threads_started":1,"wal-append.queue_time_us":101422}
I20250623 01:58:43.305361 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.326611 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.299643 14964 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.297225 14970 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:43.310045 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.311347 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.315099 13711 catalog_manager.cc:5582] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 2 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:43.318303 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.331552 14478 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:43.333559 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.335307 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.229072 14821 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:43.327239 13883 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:43.274993 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.330276 14964 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:43.321264 14926 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.329701 13882 raft_consensus.cc:2408] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:43.339422 13885 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:43.350812 13812 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:43.307636 14979 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:43.332248 14970 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.355710 13880 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:43.352658 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.353112 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.356431 13886 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:43.356591 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.356942 13882 raft_consensus.cc:2391] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:43.353969 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.268822 14968 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:43.357926 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.358670 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.359110 13882 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:43.360635 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.361016 13883 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:43.361907 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.362461 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.362602 13883 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:43.362584 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.371479 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.372967 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:43.373659 15004 log.cc:927] Time spent T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20250623 01:58:43.374229 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.375012 13879 raft_consensus.cc:2366] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.375761 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.376698 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.380295 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.381551 13885 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:43.382126 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.383064 13880 raft_consensus.cc:2366] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.383828 13854 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.385537 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.387164 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.389811 13883 raft_consensus.cc:2391] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:43.389417 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.377144 13953 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:43.393661 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:43.394364 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.387769 13711 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:43.401639 13880 raft_consensus.cc:2391] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:43.388603 14979 raft_consensus.cc:2747] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:43.389358 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:43.416610 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.421888 13711 catalog_manager.cc:5582] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:43.398798 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:43.403079 14764 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250623 01:58:43.417546 15006 log.cc:927] Time spent T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20250623 01:58:43.423805 13854 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.423707 13882 raft_consensus.cc:2366] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.428434 14478 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:43.432672 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.433595 14979 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.434098 14979 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.434440 14979 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.433552 14886 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.433048 13711 catalog_manager.cc:5582] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 3, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 3 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:43.436867 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.428026 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.443468 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.433487 15012 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:43.432816 14893 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250623 01:58:43.446373 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 177055us
I20250623 01:58:43.438282 14886 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.451812 13779 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:43.453836 15010 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:43.455106 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:43.450062 14882 raft_consensus.cc:695] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
W20250623 01:58:43.458014 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 202611us
I20250623 01:58:43.459107 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.460165 13882 raft_consensus.cc:2366] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.446393 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:43.448755 14893 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:43.468338 13776 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.460951 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.471261 13778 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:43.471426 13882 raft_consensus.cc:2366] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.459090 14886 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.473807 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:43.461517 14968 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:43.463128 13879 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:43.480069 15012 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.479957 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:43.480556 15012 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.480808 13879 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.483804 13776 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:43.485474 14775 consensus_meta.cc:287] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.533s user 0.006s sys 0.007s
I20250623 01:58:43.485699 13776 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.486855 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:43.487232 13810 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.487571 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.475385 13882 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.458907 15010 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:43.478719 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.485872 14775 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.498072 13808 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:43.492332 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.499419 14775 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.499926 13883 raft_consensus.cc:2366] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.500048 14775 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:43.495450 14764 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.496479 15010 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.504261 13879 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:43.497134 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.505661 13880 raft_consensus.cc:2366] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.472590 14893 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.472384 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.480948 15012 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.515550 15017 consensus_queue.cc:1046] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:58:43.519445 15015 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.520078 15015 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.520574 15015 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:43.515304 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 80256.2us
I20250623 01:58:43.522799 13854 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.524891 14775 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.525492 14775 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:58:43.525841 15008 log.cc:927] Time spent T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
I20250623 01:58:43.525926 14775 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.533481 15016 raft_consensus.cc:3053] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:43.533926 15016 raft_consensus.cc:738] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:43.534188 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:43.534713 15016 consensus_queue.cc:260] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:43.536698 13887 raft_consensus.cc:2366] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.508445 14764 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.548727 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.549952 13887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 996) took 2500 ms. Trace:
I20250623 01:58:43.550678 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 997) took 2500 ms. Trace:
I20250623 01:58:43.550357 13887 rpcz_store.cc:276] 0623 01:58:41.049810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.549054 (+2499244us) service_pool.cc:225] Handling call
0623 01:58:43.549407 (+ 353us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.549608 (+ 201us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:43.549632 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:43.549678 (+ 46us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:43.549716 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.549783 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:43.549886 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:43.551208 13881 rpcz_store.cc:276] 0623 01:58:41.050108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.549669 (+2499561us) service_pool.cc:225] Handling call
0623 01:58:43.550000 (+ 331us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.550192 (+ 192us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:43.550236 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:43.550304 (+ 68us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:43.550349 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.550408 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:43.550602 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:43.578694 15014 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:43.579001 14882 consensus_queue.cc:237] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.583995 13925 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.585084 14804 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.585347 13925 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.585570 14804 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.586040 14804 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.587496 14922 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:43.587846 15018 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:43.587914 14922 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:43.588539 15018 raft_consensus.cc:695] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:43.589612 15018 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.588614 14922 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.529946 14821 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.515019 14970 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.605731 13954 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
W20250623 01:58:43.629415 14481 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:43.636502 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 124989us
I20250623 01:58:43.642091 14979 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.642997 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:43.643469 14893 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:43.644203 14979 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.644814 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 423) took 1110 ms. Trace:
I20250623 01:58:43.650517 13925 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.660567 14967 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.660952 14967 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:43.661325 14967 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.645201 13884 rpcz_store.cc:276] 0623 01:58:42.534894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.643900 (+1109006us) service_pool.cc:225] Handling call
0623 01:58:43.644220 (+ 320us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.644466 (+ 246us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:43.644496 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:43.644535 (+ 39us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:43.644586 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.644653 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:43.644765 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:43.670094 14893 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.671277 14804 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.673254 14804 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:43.674587 15010 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:43.678503 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:43.679070 13954 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.678035 14764 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.646410 15012 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.695048 14969 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.698001 14775 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:43.699903 14970 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.700320 14775 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:43.701892 13925 leader_election.cc:400] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.702209 13925 leader_election.cc:403] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:43.703007 13925 leader_election.cc:400] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.703274 13925 leader_election.cc:403] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:43.704028 15010 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.706569 15012 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.690706 13954 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:43.709107 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:43.710224 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 60598.7us
I20250623 01:58:43.712265 13779 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:43.714185 14926 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.716635 14926 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.719596 15015 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.728104 14967 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.729085 14886 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.730293 13779 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:43.730573 14969 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.731223 14969 raft_consensus.cc:2762] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250623 01:58:43.711858 14922 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.735411 13852 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:43.739084 14922 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:43.735244 14804 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.736562 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 425) took 1200 ms. Trace:
I20250623 01:58:43.738974 13852 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:43.735487 13886 raft_consensus.cc:2366] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:43.731343 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:58:43.691653 15024 log.cc:927] Time spent T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.084s user 0.001s sys 0.008s
I20250623 01:58:43.743038 14970 raft_consensus.cc:2747] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:43.741986 15018 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:43.744657 15015 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:43.744807 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:43.740938 13884 rpcz_store.cc:276] 0623 01:58:42.535610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.682499 (+1146889us) service_pool.cc:225] Handling call
0623 01:58:43.735935 (+ 53436us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.736125 (+ 190us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:43.736157 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:43.736231 (+ 74us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:43.736259 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.736321 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:43.736459 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:43.745064 14804 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.750365 13884 raft_consensus.cc:1273] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:43.751102 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 427) took 1213 ms. Trace:
I20250623 01:58:43.751475 13884 rpcz_store.cc:276] 0623 01:58:42.537566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.749710 (+1212144us) service_pool.cc:225] Handling call
0623 01:58:43.750033 (+ 323us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.750799 (+ 766us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.751013 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:43.740377 15012 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:43.745199 14969 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:43.752444 15012 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:43.752562 14969 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:43.748602 14886 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.753041 15012 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.740442 14964 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.753213 14969 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.750087 13779 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:43.770473 13852 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:43.772327 13779 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:43.775499 13808 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:43.775840 13812 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:43.776281 15018 raft_consensus.cc:2802] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:43.776568 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.776726 15018 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:43.777248 15018 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:43.777312 13812 raft_consensus.cc:1273] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:43.778245 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.779392 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.779914 13852 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:43.781952 13852 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.782070 15022 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.782498 15022 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.782866 15022 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.784508 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.786517 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.787083 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.787693 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.792608 13808 raft_consensus.cc:1238] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:43.789824 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.790356 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.800235 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.802572 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.803874 13807 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:43.804669 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.805233 13807 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:43.806056 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.803246 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.801543 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.814234 13808 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:43.815126 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.817226 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.790349 15018 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:43.818320 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:43.815954 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.819491 13812 raft_consensus.cc:2366] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.820261 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.821282 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.823134 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.821897 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.826367 13807 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:43.827203 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.828405 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:43.829519 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.822309 13808 raft_consensus.cc:2366] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.830536 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.831519 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.832477 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.833406 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:43.836122 13808 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.821372 15018 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:43.824319 15022 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:43.836727 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.838621 13958 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.825670 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.840032 15022 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:43.840103 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.782821 13852 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.838954 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.842621 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.844135 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:43.843151 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 60428us
I20250623 01:58:43.844301 13882 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:43.845986 13879 raft_consensus.cc:1238] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:43.849020 13879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 985) took 3396 ms. Trace:
I20250623 01:58:43.850617 13958 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:43.851917 15023 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.851549 13879 rpcz_store.cc:276] 0623 01:58:40.453101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.513304 (+3060203us) service_pool.cc:225] Handling call
0623 01:58:43.845811 (+332507us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.846335 (+ 524us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.848827 (+ 2492us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b54003d72d0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:43.848947 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":605920128}
I20250623 01:58:43.853056 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.853885 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1010) took 1012 ms. Trace:
I20250623 01:58:43.854846 13882 rpcz_store.cc:276] 0623 01:58:42.841742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.850918 (+1009176us) service_pool.cc:225] Handling call
0623 01:58:43.853141 (+ 2223us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.853598 (+ 457us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:43.853624 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:43.853664 (+ 40us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:43.853690 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.853737 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:43.853815 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:43.856076 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 429) took 1317 ms. Trace:
I20250623 01:58:43.858026 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.858904 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.858340 13884 rpcz_store.cc:276] 0623 01:58:42.538327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.752262 (+1213935us) service_pool.cc:225] Handling call
0623 01:58:43.771398 (+ 19136us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.771692 (+ 294us) raft_consensus.cc:1532] Early marking committed up to index 2
0623 01:58:43.794034 (+ 22342us) log.cc:844] Serialized 14 byte log entry
0623 01:58:43.851596 (+ 57562us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:43.851677 (+ 81us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:43.851732 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:43.855871 (+ 4139us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:43.855997 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":22315,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":89,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":78638,"threads_started":2,"wal-append.queue_time_us":61733}
I20250623 01:58:43.860244 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:43.862536 13958 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:43.859088 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.863468 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.865409 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.860961 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.866433 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.866812 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:43.867321 13960 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.869481 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:43.873919 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:43.875226 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:43.876796 13884 raft_consensus.cc:1238] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:43.875270 13779 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:43.877899 15021 raft_consensus.cc:2762] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250623 01:58:43.878472 13884 raft_consensus.cc:1238] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:43.878592 15021 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:43.879196 15021 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:43.880648 14478 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:43.881858 15021 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.882674 13884 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:43.883195 13852 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:43.844702 15022 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.883905 15022 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:43.884310 15022 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.883968 14979 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:43.882546 13960 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:43.885972 15018 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:43.886358 14478 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:43.886487 15018 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:43.886680 15020 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:43.887142 15018 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:43.887998 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.889146 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.889796 13852 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:43.885622 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.891760 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.893086 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.894374 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.892761 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.905179 13807 raft_consensus.cc:2391] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:43.892761 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:43.910068 13960 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
W20250623 01:58:43.917701 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 59289.9us
I20250623 01:58:43.919732 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.920809 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.921291 13961 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:43.921984 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.922832 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.923306 13961 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:43.923988 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.924557 13808 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:43.924875 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:43.925792 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.926775 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.927736 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.930480 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:43.935673 13808 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:43.933375 15022 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.897068 14969 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.932626 15012 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:43.942565 14969 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.948664 13810 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:43.952353 15039 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:43.953922 15039 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:43.954461 15012 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:43.955668 13960 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:43.959678 15038 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:43.960078 13960 raft_consensus.cc:1238] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:43.957643 13926 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:43.960654 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 71258.7us
I20250623 01:58:43.918709 15034 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250623 01:58:43.965615 13804 raft_consensus.cc:1238] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:43.977111 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:44.292025 13806 raft_consensus.cc:2366] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:43.979516 14985 raft_consensus.cc:2802] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:43.981128 15022 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:44.336092 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:44.161929 14481 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:44.333926 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 109397us
I20250623 01:58:44.350189 13960 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:44.410494 13852 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:44.332933 15039 consensus_queue.cc:1046] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.116s
I20250623 01:58:44.449702 15039 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.450475 15039 raft_consensus.cc:3053] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:44.450834 15039 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:44.451870 15039 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.454444 13960 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:44.372267 14985 raft_consensus.cc:695] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:44.461551 14985 consensus_queue.cc:237] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:44.445938 15018 consensus_meta.cc:287] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.558s user 0.011s sys 0.000s
I20250623 01:58:44.464656 14964 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:44.465095 15018 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.466974 13879 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:44.468642 15018 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:44.506337 15039 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:44.523329 15045 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:44.524112 15045 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:44.525219 15045 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:44.530753 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 120405us
W20250623 01:58:44.631743 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 51133.9us
W20250623 01:58:44.633615 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 101911us
I20250623 01:58:44.670774 15013 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250623 01:58:44.703311 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 64105.3us
I20250623 01:58:44.704480 13852 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:44.706414 15038 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:44.707139 15038 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:44.707701 15038 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
W20250623 01:58:44.716969 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 759374us
I20250623 01:58:44.718618 13926 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:44.719744 13926 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.722245 14967 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:44.723373 13812 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.013s sys 0.001s
W20250623 01:58:44.723733 13812 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s user 0.013s sys 0.001s
I20250623 01:58:44.728487 13852 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:44.730144 13852 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:44.729161 14964 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.730664 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 476) took 1289 ms. Trace:
I20250623 01:58:44.731681 14964 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:44.732375 14964 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:44.732839 14964 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:44.734100 13882 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.005s sys 0.024s
W20250623 01:58:44.734427 13882 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.785s user 0.005s sys 0.024s
I20250623 01:58:44.734900 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 37) took 1676 ms. Trace:
I20250623 01:58:44.733019 15038 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.735124 13882 rpcz_store.cc:276] 0623 01:58:43.058130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.878836 (+820706us) service_pool.cc:225] Handling call
0623 01:58:43.879078 (+ 242us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:43.879288 (+ 210us) raft_consensus.cc:1532] Early marking committed up to index 5
0623 01:58:43.879661 (+ 373us) log.cc:844] Serialized 14 byte log entry
0623 01:58:44.734689 (+855028us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:44.734725 (+ 36us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:58:44.734750 (+ 25us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:44.734790 (+ 40us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:44.734864 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":164,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":402349,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":119,"thread_start_us":854576,"threads_started":2,"wal-append.queue_time_us":59428}
I20250623 01:58:44.737130 13882 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250623 01:58:44.737396 15038 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:44.738292 14968 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:58:44.738587 15040 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:44.739080 15040 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:44.739490 15040 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:44.738776 13852 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:44.743597 15044 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.744135 15038 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:44.744565 15038 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:44.745034 15038 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:44.745412 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:44.746524 13925 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:44.748123 13925 leader_election.cc:400] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.748431 13925 leader_election.cc:403] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:44.749078 13925 leader_election.cc:400] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.749272 15040 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.749651 15044 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:44.750095 15044 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:44.750175 15039 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.750586 15044 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:44.745265 14964 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.758138 14964 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:44.757913 15044 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:44.758477 13881 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250623 01:58:44.758970 13960 raft_consensus.cc:1238] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:44.759164 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 32) took 3123 ms. Trace:
I20250623 01:58:44.759516 13881 rpcz_store.cc:276] 0623 01:58:41.637032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.552163 (+1915131us) service_pool.cc:225] Handling call
0623 01:58:43.552506 (+ 343us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:44.758867 (+1206361us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:44.759076 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:44.760629 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:44.761046 14968 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:44.763411 15040 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:44.763381 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:44.763736 14968 consensus_queue.cc:1046] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:58:44.759982 13925 leader_election.cc:403] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:44.765637 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:44.776604 14949 raft_consensus.cc:3053] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:44.777153 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:44.778017 14949 raft_consensus.cc:738] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
W20250623 01:58:44.768965 13909 debug-util.cc:398] Leaking SignalData structure 0x7b0800092940 after lost signal to thread 12639
I20250623 01:58:44.770907 15038 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.779059 14949 consensus_queue.cc:260] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:44.779449 15039 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:44.780303 15038 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:44.773952 14985 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:44.780992 14985 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:44.731308 13884 rpcz_store.cc:276] 0623 01:58:43.441436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.909231 (+467795us) service_pool.cc:225] Handling call
0623 01:58:43.909690 (+ 459us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:44.218953 (+309263us) raft_consensus.cc:1532] Early marking committed up to index 2
0623 01:58:44.486073 (+267120us) log.cc:844] Serialized 14 byte log entry
0623 01:58:44.730121 (+244048us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:44.730202 (+ 81us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:44.730271 (+ 69us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:44.730373 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:44.730549 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":104976,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":428,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"thread_start_us":243482,"threads_started":1,"wal-append.queue_time_us":241984}
I20250623 01:58:44.773511 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:44.778147 13881 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:44.782174 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:44.782924 14985 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:44.764483 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 44794.9us
I20250623 01:58:44.785053 13926 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:44.775795 15044 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:44.785945 14967 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:44.786378 14967 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:44.787673 14967 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:44.788796 13881 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:44.787185 13926 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.778461 15040 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.789705 15040 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.782958 13884 raft_consensus.cc:2391] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:44.790829 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:44.794302 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:44.794667 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:44.792342 13925 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.795774 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:44.794934 13953 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:44.791036 13884 raft_consensus.cc:1238] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:44.796834 13925 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:44.780386 14949 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:44.800355 13880 raft_consensus.cc:1238] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:44.800460 15050 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:44.800961 13880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 989) took 3999 ms. Trace:
I20250623 01:58:44.792362 15044 consensus_queue.cc:1046] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:44.797427 15040 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.806082 13880 rpcz_store.cc:276] 0623 01:58:40.801999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:44.799863 (+3997864us) service_pool.cc:225] Handling call
0623 01:58:44.800182 (+ 319us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:44.800750 (+ 568us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:44.800906 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:44.806640 15040 consensus_queue.cc:1046] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:44.807209 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:44.807765 15040 raft_consensus.cc:3053] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:44.808224 15040 raft_consensus.cc:738] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:44.808463 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:44.803377 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:44.809561 13955 raft_consensus.cc:2374] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3. Re-sending same reply.
I20250623 01:58:44.809870 14764 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:44.809242 15040 consensus_queue.cc:260] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:44.791805 13926 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.793424 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 449) took 1013 ms. Trace:
I20250623 01:58:44.811122 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:44.811697 13956 raft_consensus.cc:2374] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3. Re-sending same reply.
I20250623 01:58:44.811475 13804 rpcz_store.cc:276] 0623 01:58:43.779623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.779736 (+ 113us) service_pool.cc:225] Handling call
0623 01:58:43.950768 (+171032us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:44.337179 (+386411us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:44.793236 (+456057us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54001dc750. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:44.793368 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":390916352}
I20250623 01:58:44.813463 15050 raft_consensus.cc:695] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:44.814718 15050 consensus_queue.cc:237] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.816334 15054 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.817025 15054 raft_consensus.cc:3053] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:44.817397 15054 raft_consensus.cc:738] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:44.813750 13926 leader_election.cc:400] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.818205 13926 leader_election.cc:403] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:44.818415 15054 consensus_queue.cc:260] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:44.819586 15044 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.819103 13926 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.821134 15044 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.822443 14967 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.827255 14479 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:44.828164 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:44.828315 14949 raft_consensus.cc:2802] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:44.829522 14967 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:44.831380 14949 raft_consensus.cc:695] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:44.835844 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:44.836161 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:44.837219 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:44.837814 13812 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:44.834834 14949 consensus_queue.cc:237] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:44.839296 14479 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.840273 14849 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.821373 13926 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.842227 13926 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.820255 15054 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:44.842907 14922 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:44.845571 13926 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:44.845799 14922 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.846640 14922 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:44.846841 13926 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:44.846982 14922 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:44.847631 14922 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:44.848624 13926 leader_election.cc:400] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.848940 13926 leader_election.cc:403] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250623 01:58:44.849912 15044 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.849861 13926 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.851130 13926 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:58:44.851980 15011 log.cc:927] Time spent T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 1.404s user 0.009s sys 0.016s
I20250623 01:58:44.852360 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.843247 14479 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:44.853395 13926 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.853819 14968 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:44.854211 14968 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:44.854521 13926 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.850780 15044 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.855525 15044 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.856246 15044 raft_consensus.cc:2747] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:44.856781 13881 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:44.856918 15044 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.854539 14968 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:44.860383 14967 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.861131 14967 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.858969 15013 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250623 01:58:44.862423 13883 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.256s user 0.006s sys 0.014s
W20250623 01:58:44.862803 13883 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.256s user 0.006s sys 0.014s
I20250623 01:58:44.863435 13883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 30) took 4669 ms. Trace:
I20250623 01:58:44.863750 13883 rpcz_store.cc:276] 0623 01:58:40.194197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.500528 (+3306331us) service_pool.cc:225] Handling call
0623 01:58:43.506083 (+ 5555us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:43.513683 (+ 7600us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:43.513710 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:43.555663 (+ 41953us) log.cc:844] Serialized 26 byte log entry
0623 01:58:44.863132 (+1307469us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:58:44.863171 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:44.863230 (+ 59us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:44.863266 (+ 36us) raft_consensus.cc:1683] finished
0623 01:58:44.863280 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:44.863387 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":25194,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":169,"thread_start_us":1256614,"threads_started":1,"wal-append.queue_time_us":74373,"wal-append.run_cpu_time_us":1666,"wal-append.run_wall_time_us":1005086}
I20250623 01:58:44.874893 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:44.875989 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 445) took 1099 ms. Trace:
I20250623 01:58:44.876427 13811 rpcz_store.cc:276] 0623 01:58:43.776956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:44.874489 (+1097533us) service_pool.cc:225] Handling call
0623 01:58:44.875915 (+ 1426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:44.867345 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:44.882562 13954 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:44.883263 13954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 993) took 1072 ms. Trace:
I20250623 01:58:44.884050 13954 rpcz_store.cc:276] 0623 01:58:43.810721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:44.866888 (+1056167us) service_pool.cc:225] Handling call
0623 01:58:44.883197 (+ 16309us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:44.868672 13778 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.389s user 0.017s sys 0.024s
I20250623 01:58:44.890429 13888 raft_consensus.cc:2366] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 4.
W20250623 01:58:44.885548 13778 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.406s user 0.017s sys 0.026s
W20250623 01:58:44.892542 13778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 1.41266e+06us
I20250623 01:58:44.900076 14967 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:44.900660 14967 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.903733 15023 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:44.904352 15023 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:44.905189 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:44.907718 13879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 461) took 1729 ms. Trace:
I20250623 01:58:44.908205 13879 rpcz_store.cc:276] 0623 01:58:43.178134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.887223 (+709089us) service_pool.cc:225] Handling call
0623 01:58:43.887503 (+ 280us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:44.467443 (+579940us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:44.907442 (+439999us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b5400127854. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:44.907659 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1332124928}
I20250623 01:58:44.910112 14514 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:44.911347 14967 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:44.911794 15044 consensus_queue.cc:1046] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:44.913064 15044 raft_consensus.cc:3053] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:44.913128 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:44.913493 15044 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:44.913798 13881 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:44.915457 13925 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:44.917008 15043 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:44.917446 15043 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:44.919211 15044 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:44.921511 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 78365.6us
I20250623 01:58:44.924870 15043 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:44.922060 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:44.926251 13962 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:44.926703 13962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 994) took 1115 ms. Trace:
I20250623 01:58:44.926971 13962 rpcz_store.cc:276] 0623 01:58:43.811343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:44.921696 (+1110353us) service_pool.cc:225] Handling call
0623 01:58:44.926650 (+ 4954us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:44.923210 14922 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:44.929415 14922 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:44.912554 14967 consensus_queue.cc:1046] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20250623 01:58:44.931854 14967 raft_consensus.cc:3053] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:44.932410 14967 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:44.930953 15044 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:44.924175 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:44.935559 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:44.935694 13885 raft_consensus.cc:2366] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.936331 13885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 477) took 1477 ms. Trace:
I20250623 01:58:44.936754 13885 rpcz_store.cc:276] 0623 01:58:43.458625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.501376 (+ 42751us) service_pool.cc:225] Handling call
0623 01:58:44.923960 (+1422584us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:44.936264 (+ 12304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10998144}
W20250623 01:58:44.937942 13776 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.452s user 0.018s sys 0.021s
W20250623 01:58:44.938400 13776 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.452s user 0.018s sys 0.022s
W20250623 01:58:44.939158 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 1.45445e+06us
I20250623 01:58:44.949285 13925 leader_election.cc:400] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:44.949695 13925 leader_election.cc:403] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:44.949808 13776 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
W20250623 01:58:44.954428 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 99991.8us
I20250623 01:58:44.955632 13926 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.957922 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:44.958807 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:44.959447 13808 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:44.960076 13926 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:44.960130 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:44.965934 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:44.967624 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:44.968277 13808 raft_consensus.cc:2366] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3.
I20250623 01:58:44.980996 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:44.981772 13812 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:44.954383 15017 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:44.995641 13852 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:44.998288 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 48560.2us
I20250623 01:58:44.962496 14922 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:44.996065 14514 raft_consensus.cc:1238] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.034325 13926 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.996873 14804 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:44.997536 15012 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:45.001650 15054 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:45.043454 13776 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:45.043663 13852 leader_election.cc:336] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Timed out: RequestConsensusVote RPC to 127.12.87.1:34781 timed out after 1.502s (SENT)
I20250623 01:58:45.035020 15017 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.046594 15017 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.047070 15017 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.048283 13776 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.049121 13776 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:44.999154 14967 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.050127 14967 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.003978 14764 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:45.053614 15060 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:45.054061 15015 raft_consensus.cc:2762] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20250623 01:58:45.054489 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.053606 14764 raft_consensus.cc:695] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:45.055322 13881 raft_consensus.cc:2366] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:45.055637 14764 consensus_queue.cc:237] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.054533 13812 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:45.058363 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.058970 13812 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.059413 14509 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:45.060751 13927 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:45.061843 13926 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.037266 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.066170 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.057137 15017 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:45.068383 14979 consensus_queue.cc:1046] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:45.069520 13804 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:45.042187 15012 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:45.070955 15012 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.054699 15015 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:45.064853 15054 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:45.076079 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.076642 13808 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.077131 14509 raft_consensus.cc:1238] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.078265 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:45.079012 13957 raft_consensus.cc:2366] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:45.079387 13957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 1002) took 1258 ms. Trace:
I20250623 01:58:45.079711 13957 rpcz_store.cc:276] 0623 01:58:43.821277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:45.078003 (+1256726us) service_pool.cc:225] Handling call
0623 01:58:45.079329 (+ 1326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:45.082954 13812 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
W20250623 01:58:45.089071 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 45438.5us
I20250623 01:58:45.069139 15043 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.090366 13809 raft_consensus.cc:1238] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:58:45.091012 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.091585 13812 raft_consensus.cc:2391] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
W20250623 01:58:45.090013 13852 leader_election.cc:336] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Timed out: RequestConsensusVote RPC to 127.12.87.1:34781 timed out after 1.564s (SENT)
W20250623 01:58:45.069216 13909 debug-util.cc:398] Leaking SignalData structure 0x7b0800166160 after lost signal to thread 13745
I20250623 01:58:45.093430 13926 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.080578 14922 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:45.098206 13926 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:45.116688 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.117888 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.118422 13812 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.121400 13925 leader_election.cc:400] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:45.121733 13925 leader_election.cc:403] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250623 01:58:45.122445 13852 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.124541 13852 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.099509 14804 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:45.131865 13854 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.608s user 0.013s sys 0.020s
W20250623 01:58:45.132215 13854 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.609s user 0.013s sys 0.020s
I20250623 01:58:45.135263 13926 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.094400 15054 raft_consensus.cc:695] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:45.120602 15044 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:45.100196 15060 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:45.137120 13812 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:45.138738 13926 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.131902 14922 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.146138 14886 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:45.146677 14886 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.148391 14886 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.149682 15043 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:45.150113 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.150329 15016 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:45.150884 13805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 442) took 1374 ms (client timeout 1529 ms). Trace:
I20250623 01:58:45.094192 15017 consensus_queue.cc:1046] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 5, Time since last communication: 0.040s
W20250623 01:58:45.140125 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 1.61737e+06us
I20250623 01:58:45.151923 13852 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.152830 15067 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:58:45.151281 13805 rpcz_store.cc:269] 0623 01:58:43.776056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:45.149817 (+1373761us) service_pool.cc:225] Handling call
0623 01:58:45.150828 (+ 1011us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:45.135973 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.153745 13811 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.153874 15054 consensus_queue.cc:237] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:45.132606 14926 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:45.141160 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.161573 14926 raft_consensus.cc:695] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:45.161805 13813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 446) took 1384 ms. Trace:
I20250623 01:58:45.144076 15012 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.162431 14926 consensus_queue.cc:237] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.164232 15015 raft_consensus.cc:3053] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:45.164506 15012 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.164670 15015 raft_consensus.cc:738] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:45.137836 13808 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
W20250623 01:58:45.153138 13909 debug-util.cc:398] Leaking SignalData structure 0x7b08002e9a20 after lost signal to thread 13837
I20250623 01:58:45.165714 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.166358 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.166841 13885 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.162309 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:45.168079 13958 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:45.168498 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 1010) took 1341 ms. Trace:
I20250623 01:58:45.168409 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.168907 13958 rpcz_store.cc:276] 0623 01:58:43.827189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:45.162043 (+1334854us) service_pool.cc:225] Handling call
0623 01:58:45.168445 (+ 6402us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:45.169231 13909 debug-util.cc:398] Leaking SignalData structure 0x7b080026cda0 after lost signal to thread 13838
I20250623 01:58:45.169749 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:45.170413 13909 debug-util.cc:398] Leaking SignalData structure 0x7b0800324d40 after lost signal to thread 13912
I20250623 01:58:45.170809 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.171470 13804 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:45.152520 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.172313 13812 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:45.173058 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:45.140511 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 91472.5us
I20250623 01:58:45.156211 15017 consensus_queue.cc:1230] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer 7d22741cfb9b408684cb82c58241b2f5 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250623 01:58:45.136953 15038 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:45.177950 15038 raft_consensus.cc:695] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:45.178197 13926 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:45.178966 14775 raft_consensus.cc:3053] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:45.179386 14775 raft_consensus.cc:738] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:45.178857 15038 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.180119 14775 consensus_queue.cc:260] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:45.181813 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.182572 13810 raft_consensus.cc:2374] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3. Re-sending same reply.
I20250623 01:58:45.183485 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.184202 14509 raft_consensus.cc:1238] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.174379 13852 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.156879 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.157760 14968 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:45.158268 13888 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 433) took 2618 ms (client timeout 2407 ms). Trace:
I20250623 01:58:45.189708 13852 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.158761 13887 raft_consensus.cc:1238] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
W20250623 01:58:45.159373 13886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1042) took 2037 ms (client timeout 1998 ms). Trace:
I20250623 01:58:45.187001 15015 consensus_queue.cc:260] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:45.190603 13886 rpcz_store.cc:269] 0623 01:58:43.121517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.356997 (+235480us) service_pool.cc:225] Handling call
0623 01:58:45.159214 (+1802217us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b54003afa54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:45.159318 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":860518656}
I20250623 01:58:45.191553 15015 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.192008 13887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1001) took 3632 ms. Trace:
I20250623 01:58:45.160006 13854 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:45.162181 13813 rpcz_store.cc:276] 0623 01:58:43.777243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:45.136544 (+1359301us) service_pool.cc:225] Handling call
0623 01:58:45.161750 (+ 25206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:45.166656 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.194221 13808 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:45.195029 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.195546 13807 raft_consensus.cc:2366] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:45.196018 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 481) took 1392 ms. Trace:
I20250623 01:58:45.196435 14968 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:45.196331 13807 rpcz_store.cc:276] 0623 01:58:43.803226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.926519 (+123293us) service_pool.cc:225] Handling call
0623 01:58:45.156789 (+1230270us) spinlock_profiling.cc:243] Waited 7.4 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b0323e 0000000000356d09 00007f41e897bdc0 00007f41e8a427d4 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:45.195934 (+ 39145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17019136}
I20250623 01:58:45.197530 14927 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:45.198032 14927 raft_consensus.cc:695] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:58:45.189896 13888 rpcz_store.cc:269] 0623 01:58:42.539513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.303786 (+764273us) service_pool.cc:225] Handling call
0623 01:58:45.158041 (+1854255us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54003af2d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:45.158207 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3615483264}
I20250623 01:58:45.195583 13808 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:45.175554 13909 debug-util.cc:398] Leaking SignalData structure 0x7b08003406e0 after lost signal to thread 13986
W20250623 01:58:45.199808 13909 debug-util.cc:398] Leaking SignalData structure 0x7b0800061de0 after lost signal to thread 13987
I20250623 01:58:45.200248 14886 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:45.202777 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:45.204983 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:45.182554 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:44.448741 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:45.208829 13805 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250623 01:58:45.223434 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 63519.4us
I20250623 01:58:45.224618 13854 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:45.225535 14964 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:58:45.225888 13909 debug-util.cc:398] Leaking SignalData structure 0x7b080033b580 after lost signal to thread 14540
W20250623 01:58:45.226256 13909 debug-util.cc:398] Leaking SignalData structure 0x7b080033b620 after lost signal to thread 14541
I20250623 01:58:45.174553 13812 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:45.176928 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 453) took 1390 ms. Trace:
I20250623 01:58:45.227159 13809 rpcz_store.cc:276] 0623 01:58:43.786780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:45.073802 (+1287022us) service_pool.cc:225] Handling call
0623 01:58:45.088052 (+ 14250us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:45.092144 (+ 4092us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:45.176799 (+ 84655us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b54003c64d0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:45.176888 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31602944}
I20250623 01:58:45.227897 13710 catalog_manager.cc:5582] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:45.230152 14775 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.176574 13885 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:45.139444 15023 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.192376 13887 rpcz_store.cc:276] 0623 01:58:41.559430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.551842 (+1992412us) service_pool.cc:225] Handling call
0623 01:58:45.158598 (+1606756us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:45.190566 (+ 31968us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:45.191843 (+ 1277us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54003af2d0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:45.191958 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3694547712}
I20250623 01:58:45.231134 13810 raft_consensus.cc:1238] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.198992 14927 consensus_queue.cc:237] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.235098 13807 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.235574 15023 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.155452 14764 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:45.236229 13883 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:45.235965 13811 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
W20250623 01:58:45.239413 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:45.238039 13854 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:45.245206 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 68756.9us
I20250623 01:58:45.247407 13854 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.246378 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 59) took 2625 ms. Trace:
I20250623 01:58:45.247794 15045 raft_consensus.cc:2762] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250623 01:58:45.249181 14964 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:45.249734 14964 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:45.250154 14964 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.251012 13776 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.251744 15017 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:45.252036 13776 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.252727 15017 raft_consensus.cc:2762] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250623 01:58:45.245158 14764 raft_consensus.cc:695] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:58:45.248315 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 58699.4us
I20250623 01:58:45.257172 13852 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.254395 13710 catalog_manager.cc:5582] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:45.263129 14764 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.272004 13883 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250623 01:58:45.272545 13808 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:45.266745 14922 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:45.275241 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:45.278672 14968 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:45.276185 15045 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:45.280761 15045 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:45.281097 15045 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.274379 14886 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:45.266494 13710 catalog_manager.cc:5582] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:45.285704 14886 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:45.286103 14886 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.286567 13710 catalog_manager.cc:5582] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:45.266841 14964 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.290521 15067 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.290860 14964 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.270233 13806 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250623 01:58:45.253867 13711 rpcz_store.cc:276] 0623 01:58:42.617659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:42.617965 (+ 306us) service_pool.cc:225] Handling call
0623 01:58:43.929926 (+1311961us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:43.929952 (+ 26us) write_op.cc:454] Released schema lock
0623 01:58:45.242786 (+1312834us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:43.645647 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:43.657512 (+ 11865us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:43.657559 (+ 47us) write_op.cc:435] Acquired schema lock
0623 01:58:43.657590 (+ 31us) tablet.cc:660] Decoding operations
0623 01:58:43.658248 (+ 658us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:43.658335 (+ 87us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:43.658362 (+ 27us) tablet.cc:683] Acquiring locks for 4 operations
0623 01:58:43.658820 (+ 458us) tablet.cc:699] Row locks acquired
0623 01:58:43.658842 (+ 22us) write_op.cc:260] PREPARE: finished
0623 01:58:43.659006 (+ 164us) write_op.cc:270] Start()
0623 01:58:43.659153 (+ 147us) write_op.cc:276] Timestamp: P: 1750643923658944 usec, L: 0
0623 01:58:43.659215 (+ 62us) op_driver.cc:348] REPLICATION: starting
0623 01:58:43.659866 (+ 651us) log.cc:844] Serialized 1783 byte log entry
0623 01:58:43.748169 (+ 88303us) op_driver.cc:464] REPLICATION: finished
0623 01:58:43.790374 (+ 42205us) write_op.cc:301] APPLY: starting
0623 01:58:43.790497 (+ 123us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:43.790691 (+ 194us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:43.790919 (+ 228us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:43.791786 (+ 867us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:43.792124 (+ 338us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0623 01:58:43.792218 (+ 94us) write_op.cc:312] APPLY: finished
0623 01:58:43.792639 (+ 421us) log.cc:844] Serialized 40 byte log entry
0623 01:58:43.820663 (+ 28024us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:43.820779 (+ 116us) write_op.cc:454] Released schema lock
0623 01:58:43.820837 (+ 58us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":40560,"apply.run_cpu_time_us":3061,"apply.run_wall_time_us":32154,"num_ops":4,"prepare.queue_time_us":189427,"prepare.run_cpu_time_us":11071,"prepare.run_wall_time_us":63006,"raft.queue_time_us":16875,"raft.run_cpu_time_us":1790,"raft.run_wall_time_us":42448,"replication_time_us":88796,"spinlock_wait_cycles":1405312,"thread_start_us":549204,"threads_started":4,"wal-append.queue_time_us":32240,"wal-append.run_cpu_time_us":5449,"wal-append.run_wall_time_us":1101333}]]}
W20250623 01:58:45.291777 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 113656us
I20250623 01:58:45.292655 14479 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:45.293503 14968 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:45.294100 14968 raft_consensus.cc:695] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:45.295943 13926 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.288781 13710 catalog_manager.cc:5582] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:45.303191 14984 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:58:45.303303 13883 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:45.306046 13854 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:45.308633 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:45.295229 15034 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election won for term 3
W20250623 01:58:45.310070 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 124287us
I20250623 01:58:45.311256 14926 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:45.313283 15040 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.313694 15067 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:45.300439 15023 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:45.317363 14926 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:45.323865 14926 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.316016 14804 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:58:45.314121 15039 consensus_queue.cc:1162] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7d22741cfb9b408684cb82c58241b2f5" responder_term: 2 status { last_received { term: 2 index: 4 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250623 01:58:45.314966 14970 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:45.327823 14970 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:45.328342 14970 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.294881 14968 consensus_queue.cc:237] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:45.327167 13813 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250623 01:58:45.336207 13854 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:45.341892 15039 consensus_queue.cc:1046] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:45.333842 15045 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.342665 15039 raft_consensus.cc:3053] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:45.342978 15039 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:45.343724 15039 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.344348 15045 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.344821 15039 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.334008 15034 raft_consensus.cc:695] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:45.346280 15034 consensus_queue.cc:237] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.348644 15065 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:45.348582 14926 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.338122 15023 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:45.352365 14926 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.355564 13776 leader_election.cc:400] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:45.355970 13776 leader_election.cc:403] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:45.340497 15029 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:45.360387 15029 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.361073 15012 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:45.361863 15012 consensus_queue.cc:1046] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:45.362895 15016 raft_consensus.cc:3053] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:45.363245 15016 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:45.363395 14821 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20250623 01:58:45.365815 13887 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:45.365267 15029 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:45.363996 15016 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.368255 15016 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.369446 13926 leader_election.cc:400] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:45.371160 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:45.372435 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.373457 13885 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:58:45.374958 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 73842.9us
I20250623 01:58:45.381280 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.382230 13885 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:45.384241 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.385691 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.386932 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.388249 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.390990 13886 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:45.392729 13882 raft_consensus.cc:2391] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:45.393986 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.394423 13887 raft_consensus.cc:1238] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:45.395160 13886 raft_consensus.cc:1273] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:45.410972 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:45.411651 13954 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.414748 13959 raft_consensus.cc:1238] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:45.410729 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:45.414800 13880 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:45.418023 13959 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:45.421919 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:45.381662 15029 consensus_queue.cc:1046] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20250623 01:58:45.425314 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:45.425611 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:45.426257 13955 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.380174 15015 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:58:45.428440 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 90066.8us
I20250623 01:58:45.381273 15023 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:45.380633 14775 consensus_queue.cc:1046] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20250623 01:58:45.436551 13960 raft_consensus.cc:1238] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.437924 13954 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:45.383370 15012 consensus_queue.cc:1046] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20250623 01:58:45.425956 13953 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.469724 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:45.470443 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 65445.6us
I20250623 01:58:45.391260 14886 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.434358 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:45.487136 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:45.490335 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 61293.5us
I20250623 01:58:45.431869 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:45.473373 15074 log.cc:927] Time spent T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20250623 01:58:45.438581 14882 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:45.493852 13958 raft_consensus.cc:2408] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:45.435588 15015 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:45.495335 13959 raft_consensus.cc:1273] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:45.496177 13854 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:45.501434 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.503484 13807 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.501704 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.501000 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.505481 13808 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.506034 13888 raft_consensus.cc:1238] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:45.506160 13882 raft_consensus.cc:1238] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250623 01:58:45.506551 13961 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250623 01:58:45.507416 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:45.507278 13925 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.434698 15017 consensus_queue.cc:1046] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20250623 01:58:45.432096 15065 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20250623 01:58:45.524012 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:45.526552 13887 raft_consensus.cc:1238] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:45.530582 15029 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:45.531068 15029 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.532696 15015 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:45.439500 15023 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:45.532780 15029 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:45.541414 13808 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
W20250623 01:58:45.544818 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 110731us
I20250623 01:58:45.495937 15038 consensus_queue.cc:1046] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20250623 01:58:45.530082 15039 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:45.555788 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:45.550457 13852 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:45.555757 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 59717.8us
I20250623 01:58:45.513866 14882 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:45.601576 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 43785.3us
I20250623 01:58:45.613582 15039 raft_consensus.cc:695] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:58:45.595224 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 210051us
I20250623 01:58:45.529381 15034 consensus_queue.cc:1046] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20250623 01:58:45.593406 13882 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:45.496351 15082 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20250623 01:58:45.546180 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.616771 13854 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:45.618283 13808 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250623 01:58:45.556283 15067 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:45.559816 15073 raft_consensus.cc:2762] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250623 01:58:45.620600 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:45.620555 13779 leader_election.cc:336] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.603s (SENT)
I20250623 01:58:45.622200 13807 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
W20250623 01:58:45.617897 15077 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20250623 01:58:45.556479 14926 consensus_queue.cc:1046] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20250623 01:58:45.635718 13961 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:45.557468 14775 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20250623 01:58:45.637821 14967 raft_consensus.cc:3053] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
W20250623 01:58:45.640101 15080 log.cc:927] Time spent T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250623 01:58:45.641129 14984 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:45.595985 14922 consensus_queue.cc:1046] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20250623 01:58:45.642216 15075 raft_consensus.cc:3053] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:45.643025 13954 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250623 01:58:45.621193 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:45.592798 14804 consensus_queue.cc:1046] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20250623 01:58:45.634682 13811 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:45.658638 13779 leader_election.cc:336] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.705s (SENT)
I20250623 01:58:45.659823 13879 raft_consensus.cc:1238] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.663322 13807 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:45.666366 14509 raft_consensus.cc:2391] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:45.667461 14804 consensus_queue.cc:1230] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250623 01:58:45.668063 14970 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.672093 13880 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250623 01:58:45.653910 15075 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:45.673405 15075 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.674990 15075 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.645294 14967 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:45.669113 15040 raft_consensus.cc:3053] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:45.696465 14882 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.696650 15040 raft_consensus.cc:738] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:45.697504 15040 consensus_queue.cc:260] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.698644 15040 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.700851 15087 raft_consensus.cc:3053] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:45.701224 15087 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:45.701812 13953 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:45.701936 15087 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.703011 15087 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.636318 15067 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.705090 15034 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:45.705464 15067 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:45.705605 15034 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.619134 15039 consensus_queue.cc:237] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.688782 15081 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:45.691437 14927 raft_consensus.cc:3053] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:45.691531 15023 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:45.692747 15038 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:45.647082 15088 consensus_queue.cc:1046] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20250623 01:58:45.694681 14967 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.696135 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:45.685881 14886 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.712523 13883 raft_consensus.cc:1238] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250623 01:58:45.724826 13955 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
W20250623 01:58:45.744788 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:45.752995 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 103831us
W20250623 01:58:45.761109 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 210733us
W20250623 01:58:45.761535 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:45.766763 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:45.813479 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.814607 13808 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:58:45.815220 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.815999 13810 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:58:45.818464 13808 raft_consensus.cc:1238] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.819378 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.713482 14927 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:45.822780 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.713755 15038 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:45.830621 13806 raft_consensus.cc:1238] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20250623 01:58:45.831826 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 215129us
I20250623 01:58:45.816942 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.823522 13809 raft_consensus.cc:2391] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:45.821038 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.760039 13884 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.756704 15081 raft_consensus.cc:695] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:45.839542 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.827714 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.810245 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:45.835889 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.846284 13808 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:45.841910 13812 raft_consensus.cc:2366] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:58:45.851518 13806 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:45.851622 13779 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:45.833171 15038 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:45.824038 14984 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20250623 01:58:45.813376 15016 consensus_queue.cc:1046] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20250623 01:58:45.857661 15034 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:45.859903 14927 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.862605 15015 raft_consensus.cc:3053] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:45.906101 15015 raft_consensus.cc:738] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:45.906972 15015 consensus_queue.cc:260] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.866165 15065 raft_consensus.cc:3053] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:45.866547 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:45.867594 15029 raft_consensus.cc:3053] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:45.909029 15012 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:45.868927 14970 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:45.910022 14882 raft_consensus.cc:3053] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:45.869151 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:45.871507 14922 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:45.831745 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:58:45.879348 15092 log.cc:927] Time spent T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
W20250623 01:58:45.884511 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:45.902590 14967 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.898550 13888 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.899230 13852 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:45.900076 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:45.917883 14882 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:45.924535 15096 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:45.922537 14922 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.925992 15096 raft_consensus.cc:695] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:45.925941 14882 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.927057 15096 consensus_queue.cc:237] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.927552 14882 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.898299 14979 raft_consensus.cc:3053] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:45.929507 14979 raft_consensus.cc:738] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:45.909251 15029 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:45.930529 14979 consensus_queue.cc:260] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.868052 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:45.937852 13852 leader_election.cc:400] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:58:45.938162 13852 leader_election.cc:403] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20250623 01:58:45.911137 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 44690.2us
I20250623 01:58:45.911302 13806 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:45.939306 13854 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:45.911607 15073 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:45.912072 15015 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.911976 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:45.888564 15088 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:45.917268 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 70001.4us
I20250623 01:58:45.945742 13779 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.894769 13883 raft_consensus.cc:1238] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:45.901587 13886 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250623 01:58:45.932978 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:45.822270 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 65897.5us
I20250623 01:58:45.909055 15065 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:45.955364 13854 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:45.931818 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:45.953532 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:45.956835 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 127666us
I20250623 01:58:45.959525 13885 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:45.927574 14922 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:45.981736 13779 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:45.985606 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.943571 15082 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 4
I20250623 01:58:45.963387 13854 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:45.991742 13809 raft_consensus.cc:1273] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:45.992019 13779 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:45.946427 14775 raft_consensus.cc:3053] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:46.005945 14775 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:46.006932 14775 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.008404 14775 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.012967 15065 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.016112 13811 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:45.957290 15097 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20250623 01:58:45.990414 15088 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.023144 15054 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:46.023761 15054 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.931133 15029 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.025815 15029 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.957594 13804 raft_consensus.cc:2366] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
W20250623 01:58:46.030779 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:46.031345 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.032922 15098 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:46.034288 15012 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:46.034889 15012 raft_consensus.cc:695] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:46.035949 15012 consensus_queue.cc:237] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:46.037091 15017 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:58:46.038763 15089 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.216s user 0.002s sys 0.000s
I20250623 01:58:45.954726 15095 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:46.040519 15095 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:46.040956 15095 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:45.997342 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:46.001645 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:46.044924 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:45.920192 15081 consensus_queue.cc:237] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:45.956835 13879 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:46.044974 13883 raft_consensus.cc:2391] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:58:46.052860 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 89566.4us
I20250623 01:58:46.073971 13888 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:46.077776 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.083981 13956 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.078632 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.078233 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.086371 13958 raft_consensus.cc:2391] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:46.081065 13813 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:46.080087 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.095419 13959 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:46.096436 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.102409 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:46.103272 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 111323us
I20250623 01:58:46.104198 13779 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.014506 15065 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:46.104545 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.113617 13958 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:46.101147 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:46.111987 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.119402 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.071636 15094 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:46.102797 13881 raft_consensus.cc:2391] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:45.990039 15034 consensus_queue.cc:1046] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20250623 01:58:46.128407 13956 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:46.097970 15017 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.134724 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.139400 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.110479 14926 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:46.074525 15040 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20250623 01:58:46.141337 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.142118 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.069449 14922 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20250623 01:58:46.121086 15088 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.146546 13779 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.152050 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:46.144614 13956 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:46.144887 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 149278us
I20250623 01:58:46.157871 13852 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:46.161527 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.141985 15094 raft_consensus.cc:695] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:46.172505 15094 consensus_queue.cc:237] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.174201 15088 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.171209 15083 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:46.175532 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.171506 15084 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:46.177491 13779 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:46.170181 13959 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250623 01:58:46.180047 13957 consensus_meta.cc:287] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.667s user 0.005s sys 0.005s
I20250623 01:58:46.180644 14509 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:46.151825 15017 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.190709 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.191843 13959 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:46.192912 13886 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:46.153537 15054 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.197960 13779 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.190462 13957 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:46.210548 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.215034 14979 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.215363 13953 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:46.218142 14927 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.220484 15039 raft_consensus.cc:3053] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:46.220996 15039 raft_consensus.cc:738] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:46.222128 15039 consensus_queue.cc:260] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.223695 15039 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.225114 13805 raft_consensus.cc:2408] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:46.226819 13812 raft_consensus.cc:2408] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:46.236559 13779 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.204341 15016 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:46.202917 15084 raft_consensus.cc:695] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:58:46.240721 15087 consensus_meta.cc:287] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.537s user 0.011s sys 0.000s
I20250623 01:58:46.252734 13960 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:46.256914 13962 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:46.257882 13884 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:46.240531 15017 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:46.259188 13956 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:46.259478 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.263181 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.263720 13962 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:46.244400 15016 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:58:46.268674 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.270493 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.272887 13958 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:46.265754 14922 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:46.275271 14922 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.275949 13957 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:46.279523 13962 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:46.280470 13961 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:46.281702 13962 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250623 01:58:46.281596 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 123823us
I20250623 01:58:46.282490 13956 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:46.258088 15095 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:46.292034 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.299417 15084 consensus_queue.cc:237] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.284467 14926 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:58:46.302856 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.270815 15016 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.277179 14922 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.308346 13954 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:46.310950 13956 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:46.311021 13960 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:46.312731 13882 raft_consensus.cc:1238] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:46.311880 14926 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:46.330363 13925 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:46.331204 13812 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:46.334399 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.335106 13883 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:46.336864 13959 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:46.338778 14926 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:46.339417 15038 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.342056 15029 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:46.342463 15029 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:46.342777 15029 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:46.343777 13880 raft_consensus.cc:1238] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:46.353672 15095 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:46.354283 15095 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:46.358937 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:46.359230 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:46.359572 13810 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
W20250623 01:58:46.362146 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.362036 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:46.368294 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:46.349560 15038 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:46.369150 14509 raft_consensus.cc:2391] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 2: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:46.324241 15073 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20250623 01:58:46.360819 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:46.380681 15105 log.cc:927] Time spent T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.154s user 0.000s sys 0.002s
I20250623 01:58:46.380990 13884 raft_consensus.cc:2391] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:58:46.382890 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 146378us
W20250623 01:58:46.383154 15104 log.cc:927] Time spent T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20250623 01:58:46.383635 13779 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:58:46.386723 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 148117us
I20250623 01:58:46.387404 13776 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:46.350566 14922 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.034s
I20250623 01:58:46.389194 15012 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:46.390909 15012 raft_consensus.cc:695] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:46.393481 15012 consensus_queue.cc:237] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.377188 13953 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:46.411540 13956 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:46.411798 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:46.399015 15093 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.414127 13955 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:46.408675 15088 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:46.409960 14948 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:46.422483 15082 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:58:46.420732 15065 consensus_queue.cc:1046] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20250623 01:58:46.431591 14948 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:46.446623 13953 raft_consensus.cc:1273] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:46.451589 13882 raft_consensus.cc:1273] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250623 01:58:46.459116 15039 consensus_queue.cc:1035] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:46.459694 13808 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:46.461318 15095 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:46.465039 13888 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250623 01:58:46.481801 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:46.500406 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 92475.8us
I20250623 01:58:46.505002 13958 raft_consensus.cc:1238] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:46.507500 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.509629 13883 raft_consensus.cc:2391] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:46.512964 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:46.513568 13808 raft_consensus.cc:2391] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:46.516738 13809 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:46.515336 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:46.520216 15123 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
W20250623 01:58:46.520682 15121 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250623 01:58:46.521332 15029 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:46.521757 15124 log.cc:927] Time spent T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250623 01:58:46.525066 15029 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.529570 14479 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.530596 14893 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.519268 15095 consensus_queue.cc:1046] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20250623 01:58:46.533878 15095 raft_consensus.cc:3053] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:46.534196 15095 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:46.534981 15095 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.515605 14922 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:46.536629 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.536290 14922 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:46.537820 13958 raft_consensus.cc:2391] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 2: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:46.547168 14922 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:46.549667 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.550421 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:46.561983 13953 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:46.552840 13854 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:46.555174 15016 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.539198 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:46.566764 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 155080us
I20250623 01:58:46.559365 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.551537 13955 raft_consensus.cc:1273] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:46.565454 13854 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:46.567565 13887 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
W20250623 01:58:46.566038 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.567584 13779 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:46.572556 14893 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.570676 13854 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:46.588722 15040 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:58:46.589944 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:46.594393 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.564744 14948 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:46.568434 14927 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:46.595770 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:46.585929 14967 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:46.596426 14509 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:46.596611 14967 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.586301 15094 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:58:46.599741 13927 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:46.602195 13854 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:46.607880 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.609736 13854 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:46.598582 14967 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.619637 13854 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:46.617854 15073 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:46.620651 13953 raft_consensus.cc:2391] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:46.613674 15117 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.610962 15034 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:46.626320 15034 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:46.626657 15034 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:46.630246 15073 raft_consensus.cc:695] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:46.631135 15073 consensus_queue.cc:237] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.623692 14922 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:46.633819 14922 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.635723 14922 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.608534 15094 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:46.638393 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:46.641417 13854 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:46.642370 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.643131 13888 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:46.644156 13925 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:46.645159 15112 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.636531 15016 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:46.641232 15045 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:46.678789 15084 raft_consensus.cc:2802] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:46.646358 15094 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20250623 01:58:46.673563 13854 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:46.637928 14968 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20250623 01:58:46.683797 14948 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:46.684499 14948 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:46.686560 15039 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:46.687180 15039 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.690547 15039 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.690920 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.693789 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:46.694640 13804 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:46.680105 15029 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:46.697141 15029 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:46.697541 15029 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:46.706434 13854 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:58:46.701860 15133 log.cc:927] Time spent T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250623 01:58:46.712769 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:46.701551 14927 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:46.706410 15094 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20250623 01:58:46.709170 15075 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:46.718760 13854 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:46.719643 14927 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:46.720445 15034 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:46.720592 14927 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.723801 15034 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:46.724610 14979 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250623 01:58:46.724778 15134 log.cc:927] Time spent T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20250623 01:58:46.725057 14979 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:46.725472 14979 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.727981 15097 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:46.729415 15088 raft_consensus.cc:695] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:46.729794 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:46.730360 13887 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:46.730314 15088 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.730994 13708 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 2 to 3, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 3 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:46.732522 13708 catalog_manager.cc:5582] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:46.734033 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.734486 13886 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 1.
I20250623 01:58:46.714481 15111 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:46.754782 14478 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:46.712234 15117 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20250623 01:58:46.734236 13708 catalog_manager.cc:5582] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 3, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 3 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:46.756767 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52432 (request call id 165) took 3688 ms. Trace:
I20250623 01:58:46.759091 13887 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:46.756281 14926 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:46.709167 15084 raft_consensus.cc:695] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:46.716567 14964 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20250623 01:58:46.772495 14979 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.756418 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:46.768190 15084 consensus_queue.cc:237] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:46.768390 15029 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.754921 15075 raft_consensus.cc:695] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:46.756572 14849 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:46.757923 13708 catalog_manager.cc:5582] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 2 to 3, term changed from 1 to 2, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:46.759163 13710 rpcz_store.cc:276] 0623 01:58:43.068116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:43.068250 (+ 134us) service_pool.cc:225] Handling call
0623 01:58:46.756708 (+3688458us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:45.422884 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:45.423426 (+ 542us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:45.423457 (+ 31us) write_op.cc:435] Acquired schema lock
0623 01:58:45.423473 (+ 16us) tablet.cc:660] Decoding operations
0623 01:58:45.424098 (+ 625us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:45.424152 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:45.424177 (+ 25us) tablet.cc:683] Acquiring locks for 5 operations
0623 01:58:45.424670 (+ 493us) tablet.cc:699] Row locks acquired
0623 01:58:45.424683 (+ 13us) write_op.cc:260] PREPARE: finished
0623 01:58:45.424822 (+ 139us) write_op.cc:270] Start()
0623 01:58:45.424922 (+ 100us) write_op.cc:276] Timestamp: P: 1750643925424778 usec, L: 0
0623 01:58:45.424938 (+ 16us) op_driver.cc:348] REPLICATION: starting
0623 01:58:45.425616 (+ 678us) log.cc:844] Serialized 2063 byte log entry
0623 01:58:45.941772 (+516156us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:58:46.221875 (+280103us) op_driver.cc:464] REPLICATION: finished
0623 01:58:46.347238 (+125363us) write_op.cc:301] APPLY: starting
0623 01:58:46.352134 (+ 4896us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:46.368435 (+ 16301us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:46.368484 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:46.382279 (+ 13795us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:46.382489 (+ 210us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0623 01:58:46.382545 (+ 56us) write_op.cc:312] APPLY: finished
0623 01:58:46.382796 (+ 251us) log.cc:844] Serialized 46 byte log entry
0623 01:58:46.382970 (+ 174us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:46.383081 (+ 111us) write_op.cc:454] Released schema lock
0623 01:58:46.383130 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:46.383485 (+ 355us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:46.383518 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":417999,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":92632,"apply.run_cpu_time_us":2673,"apply.run_wall_time_us":53354,"num_ops":5,"prepare.queue_time_us":111294,"prepare.run_cpu_time_us":23653,"prepare.run_wall_time_us":172158,"raft.queue_time_us":181813,"raft.run_cpu_time_us":8552,"raft.run_wall_time_us":42932,"replication_time_us":796802,"spinlock_wait_cycles":8746752,"thread_start_us":342842,"threads_started":4,"wal-append.queue_time_us":282032}]]}
I20250623 01:58:46.756351 14948 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:46.786619 13708 catalog_manager.cc:5582] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 2, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 2 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250623 01:58:46.789355 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 91118.1us
I20250623 01:58:46.784272 14514 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:46.784881 14849 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:46.788735 14979 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:46.813252 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 303242us
I20250623 01:58:46.812211 15126 raft_consensus.cc:3053] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:46.805769 14849 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:46.816030 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:46.822252 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 69230.3us
I20250623 01:58:46.825821 15096 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:46.828085 15111 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:46.829015 14926 raft_consensus.cc:695] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:46.814019 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:46.862418 13854 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:46.863561 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:46.859807 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 292314us
I20250623 01:58:46.868341 13779 leader_election.cc:400] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:46.829356 15111 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.831566 15087 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:46.871356 13779 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:46.816917 15126 raft_consensus.cc:738] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:46.874697 15087 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:46.880960 15087 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.881300 15126 consensus_queue.cc:260] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.882738 15126 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.874729 15038 consensus_queue.cc:1046] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20250623 01:58:46.880427 15141 log.cc:927] Time spent T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.114s user 0.006s sys 0.000s
I20250623 01:58:46.886518 14886 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.888949 15150 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:46.889385 15150 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:46.889742 15150 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:46.871852 13953 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:46.886180 13854 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:46.873912 14922 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:46.829973 14926 consensus_queue.cc:237] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.896739 13779 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.898312 13779 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:46.885449 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:46.892578 15135 consensus_queue.cc:1046] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20250623 01:58:46.898551 15034 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:46.899379 14922 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.905992 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:46.906543 13887 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:46.906314 15034 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.907644 14922 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.893255 15083 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20250623 01:58:46.833470 15075 consensus_queue.cc:237] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.910748 15029 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.910998 15125 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:46.911509 15125 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:46.914968 13779 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.899966 14979 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:46.920679 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.919248 13887 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:46.928893 13925 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:46.905541 15073 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:46.935178 14514 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:46.936311 13779 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:46.926803 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:46.938046 13779 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.939658 13779 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:46.941182 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:58:46.948469 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:46.935057 15151 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:46.917454 15132 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:46.964381 13927 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:46.965363 14967 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:46.919324 15034 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:46.936110 15015 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:46.942081 15054 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:46.956694 15017 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:46.935735 14922 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:46.942297 15040 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:46.956867 15093 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:46.960244 15065 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:46.979736 13926 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.957099 15135 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250623 01:58:46.972061 14979 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:46.970425 15132 raft_consensus.cc:695] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:46.974267 15017 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:46.962888 15107 consensus_queue.cc:1046] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20250623 01:58:46.982167 15054 raft_consensus.cc:695] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:47.000541 13926 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.005630 13926 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:46.988930 15065 raft_consensus.cc:695] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:46.975502 14922 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.008172 13926 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:47.009171 15147 log.cc:927] Time spent T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20250623 01:58:46.978610 15040 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:46.994876 14979 raft_consensus.cc:695] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:46.994552 15017 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:47.017575 14979 consensus_queue.cc:237] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.018388 15095 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:47.007295 15029 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250623 01:58:47.023903 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 137850us
I20250623 01:58:47.028180 13961 raft_consensus.cc:1238] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:47.029170 13954 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250623 01:58:47.021610 15142 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20250623 01:58:47.045811 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:47.048656 13854 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
W20250623 01:58:47.055707 15119 log.cc:927] Time spent T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.608s user 0.003s sys 0.000s
I20250623 01:58:47.048816 15029 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:47.053093 14967 raft_consensus.cc:695] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:47.073949 15117 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:47.075069 14970 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:58:47.095587 15143 log.cc:927] Time spent T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20250623 01:58:47.109844 13881 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:47.084605 15081 consensus_queue.cc:1046] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250623 01:58:47.117506 13953 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:47.146824 15164 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:47.086127 15132 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.131470 15029 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:47.131880 15038 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:47.226792 15156 log.cc:927] Time spent T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20250623 01:58:47.227242 15038 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.229147 15038 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:47.231611 13882 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:47.234436 13882 raft_consensus.cc:1238] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:47.235844 15107 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:47.236080 15045 raft_consensus.cc:3053] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:47.236478 15045 raft_consensus.cc:738] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:47.237130 15107 consensus_queue.cc:1046] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:47.109606 15054 consensus_queue.cc:237] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.130724 14970 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.109283 15065 consensus_queue.cc:237] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:47.301797 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 249752us
I20250623 01:58:47.221688 15125 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:47.247383 15135 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20250623 01:58:47.286435 14922 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:47.290903 15040 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:47.291210 15096 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.237229 15045 consensus_queue.cc:260] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.331249 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.341580 13961 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:47.342751 13961 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250623 01:58:47.342897 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 334770us
I20250623 01:58:47.347594 13926 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.349277 15125 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.359695 13806 raft_consensus.cc:1273] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:47.361114 13807 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:47.361650 15093 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.362102 15093 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.364321 15093 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:47.364806 15093 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.365157 13807 raft_consensus.cc:1238] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:47.365244 15093 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:47.371239 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 403416us
I20250623 01:58:47.359994 13926 leader_election.cc:400] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:47.375617 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:47.376343 13960 raft_consensus.cc:2366] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4.
I20250623 01:58:47.377324 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.379128 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.380894 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.409421 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:47.410185 13960 raft_consensus.cc:2366] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:47.410745 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.410874 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.411752 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:47.413528 15016 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.413997 15016 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.415479 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:47.416217 15016 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:47.311306 15112 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20250623 01:58:47.416895 14979 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.417577 14979 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.419945 14979 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:47.427872 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:47.429518 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:47.437232 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.433616 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.440587 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.290417 14967 consensus_queue.cc:237] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:47.438791 13812 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:47.441314 13806 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:47.448144 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.449535 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:47.450249 13806 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:47.451350 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:47.346153 15096 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.431473 15163 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:47.459021 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.462621 13812 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:47.464478 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.349819 15125 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.436684 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:58:47.476529 15087 consensus_meta.cc:287] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.595s user 0.008s sys 0.000s
I20250623 01:58:47.477352 15045 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:47.478346 15131 raft_consensus.cc:3053] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:47.483873 14514 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:47.486425 13806 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:47.494246 15096 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:47.494601 15157 log.cc:927] Time spent T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.320s user 0.000s sys 0.001s
W20250623 01:58:47.496574 15159 log.cc:927] Time spent T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.357s user 0.002s sys 0.000s
I20250623 01:58:47.486518 15131 raft_consensus.cc:738] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:47.503513 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.489637 14970 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:47.509785 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.510543 13806 raft_consensus.cc:2408] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:47.490916 15093 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:47.497807 15168 log.cc:927] Time spent T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20250623 01:58:47.513618 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:47.499274 15150 consensus_meta.cc:287] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.609s user 0.007s sys 0.003s
I20250623 01:58:47.517022 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:47.521216 14849 consensus_meta.cc:287] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Time spent flushing consensus metadata: real 0.660s user 0.003s sys 0.006s
W20250623 01:58:47.522782 15126 consensus_meta.cc:287] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.640s user 0.011s sys 0.000s
I20250623 01:58:47.529597 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.486701 15087 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.535456 13806 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
W20250623 01:58:47.525591 15146 log.cc:927] Time spent T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.742s user 0.003s sys 0.000s
I20250623 01:58:47.538364 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.538563 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.540416 13961 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:47.542299 15125 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:47.542945 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:47.545419 15017 consensus_meta.cc:287] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.504s user 0.009s sys 0.000s
I20250623 01:58:47.545912 15017 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:47.546270 15181 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.546331 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:47.547030 13888 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:47.546881 15181 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.550325 13926 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.541208 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.551472 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:47.551715 13960 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:47.552147 13888 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:47.551664 13926 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:47.550912 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:47.553419 13926 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:47.526772 15131 consensus_queue.cc:260] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:47.554605 15170 log.cc:927] Time spent T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
I20250623 01:58:47.554674 15040 raft_consensus.cc:2762] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250623 01:58:47.555049 15131 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:47.536761 14849 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:47.560380 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:47.564815 13812 raft_consensus.cc:1238] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:47.566111 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:47.566936 14514 raft_consensus.cc:2366] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:47.569458 13813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1134) took 1099 ms. Trace:
I20250623 01:58:47.569703 13926 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.571121 13926 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:47.577798 13778 leader_election.cc:400] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Vote denied by peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393) with higher term. Message: Invalid argument: T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:47.578176 13778 leader_election.cc:403] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250623 01:58:47.573341 15176 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:47.587267 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.569901 13813 rpcz_store.cc:276] 0623 01:58:46.464256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:46.464425 (+ 169us) service_pool.cc:225] Handling call
0623 01:58:46.464818 (+ 393us) raft_consensus.cc:1469] Updating replica for 2 ops
0623 01:58:46.758239 (+293421us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:46.758282 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0623 01:58:46.913262 (+154980us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b2c000014c8. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e24426d1 00007f41e2451d02 00007f41e3adac51 00007f41e3a9f02a 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832
0623 01:58:46.913699 (+ 437us) log.cc:844] Serialized 52 byte log entry
0623 01:58:46.973959 (+ 60260us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:46.973998 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:46.974127 (+ 129us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:47.549723 (+575596us) raft_consensus.cc:1683] finished
0623 01:58:47.549740 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:47.563325 (+ 13585us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":308727,"raft.queue_time_us":336707,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":10515,"spinlock_wait_cycles":3229568,"thread_start_us":59902,"threads_started":1,"wal-append.queue_time_us":91782}
I20250623 01:58:47.563617 15125 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:47.549662 13804 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:47.559670 15040 raft_consensus.cc:2762] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250623 01:58:47.594784 13813 raft_consensus.cc:1238] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:47.558301 15017 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:47.558166 15181 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:47.534569 15150 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.589358 15126 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:47.595152 13959 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.024s sys 0.044s
W20250623 01:58:47.670672 13959 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.024s sys 0.044s
I20250623 01:58:47.599949 13813 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:47.592907 15163 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:47.595757 15125 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:47.635360 13779 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:47.636046 15082 raft_consensus.cc:3053] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:47.654314 15087 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:47.638805 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 272c03e6252d489cbf2fdceb36b05dc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:47.640789 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:47.598066 14922 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:58:47.653646 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 311597us
I20250623 01:58:47.661201 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:47.658304 13812 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:47.751689 13961 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250623 01:58:47.650210 15177 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.660913 13888 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:47.597489 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:47.756495 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 159094us
I20250623 01:58:47.757344 13926 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.758635 13959 raft_consensus.cc:1273] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:47.758905 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.759488 13961 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:47.759917 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:47.760676 13959 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:47.761886 13959 raft_consensus.cc:1238] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:47.764204 13959 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20250623 01:58:47.726683 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 91507us
I20250623 01:58:47.636785 13881 raft_consensus.cc:1238] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:47.748965 13809 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:47.651237 14849 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:47.751042 13695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 112418us
I20250623 01:58:47.767927 13779 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.743459 15125 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:47.775779 13779 leader_election.cc:400] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4.
I20250623 01:58:47.742352 15107 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:47.777959 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:47.781553 13779 leader_election.cc:403] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250623 01:58:47.770237 13812 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:47.742421 15082 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:47.786633 14922 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:47.779990 14979 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:47.787323 15087 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4.
I20250623 01:58:47.785635 13854 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:47.750628 15163 raft_consensus.cc:695] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:47.773926 15176 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:47.780925 13804 raft_consensus.cc:1273] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:47.805913 15178 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:47.807235 15150 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:47.766607 15181 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:47.812217 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:47.787856 15016 consensus_queue.cc:1046] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250623 01:58:47.774108 15177 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.807736 15082 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:47.815094 15095 consensus_meta.cc:287] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.796s user 0.004s sys 0.007s
W20250623 01:58:47.841928 13852 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.013s sys 0.022s
W20250623 01:58:47.825977 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 388224us
I20250623 01:58:47.825264 14922 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.885557 13884 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:47.843380 14970 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20250623 01:58:47.843042 14964 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:47.952903 13812 raft_consensus.cc:1273] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250623 01:58:47.953778 13812 pending_rounds.cc:85] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 1
W20250623 01:58:47.922360 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 77443.1us
W20250623 01:58:47.924420 13810 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.002s sys 0.032s
W20250623 01:58:47.954981 13810 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.596s user 0.002s sys 0.033s
I20250623 01:58:47.785197 13779 leader_election.cc:400] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:47.955928 13779 leader_election.cc:403] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20250623 01:58:47.926733 13852 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.599s user 0.013s sys 0.022s
W20250623 01:58:47.958669 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 173653us
W20250623 01:58:47.961836 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:47.959635 13779 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:47.937994 15132 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.021s sys 0.021s
W20250623 01:58:47.964733 15132 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s user 0.021s sys 0.021s
I20250623 01:58:47.965659 13812 pending_rounds.cc:107] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250623 01:58:47.972841 14514 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250623 01:58:47.974865 15095 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:47.976225 15095 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:47.976960 15038 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:47.980693 15084 consensus_queue.cc:1046] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:47.982174 15038 raft_consensus.cc:3053] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:47.982627 15038 raft_consensus.cc:738] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:47.983505 15038 consensus_queue.cc:260] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:47.985215 15038 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:47.965992 13812 raft_consensus.cc:2887] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:47.992662 13779 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.937573 15150 raft_consensus.cc:3053] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:47.919255 15181 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:47.997249 13887 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.008s sys 0.039s
I20250623 01:58:47.937898 15034 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:47.994765 13854 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:48.000093 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:47.941819 14979 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:47.951818 14964 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:48.017531 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:58:48.005760 13887 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s user 0.008s sys 0.039s
I20250623 01:58:48.013005 13854 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:48.023694 13813 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:48.025379 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.026839 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.027616 13813 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:48.029392 15016 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:47.937799 15191 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20250623 01:58:48.048164 15097 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:48.071858 13962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1120) took 1044 ms. Trace:
W20250623 01:58:48.075383 15192 log.cc:927] Time spent T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20250623 01:58:48.076553 13956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1117) took 1049 ms. Trace:
W20250623 01:58:48.076679 15172 log.cc:927] Time spent T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.375s user 0.000s sys 0.001s
I20250623 01:58:48.076925 13956 rpcz_store.cc:276] 0623 01:58:47.026699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.030079 (+ 3380us) service_pool.cc:225] Handling call
0623 01:58:47.030328 (+ 249us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:47.342288 (+311960us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:47.342317 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:47.348973 (+ 6656us) log.cc:844] Serialized 26 byte log entry
0623 01:58:47.597179 (+248206us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:47.597222 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:47.597291 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:48.076334 (+479043us) raft_consensus.cc:1683] finished
0623 01:58:48.076353 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:48.076491 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105899,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":534678,"threads_started":2,"wal-append.queue_time_us":455911}
I20250623 01:58:48.078955 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1114) took 1053 ms. Trace:
W20250623 01:58:48.079144 15180 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.445s user 0.004s sys 0.000s
I20250623 01:58:48.082556 15177 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.084578 15177 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.085161 15177 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.087409 15177 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.047683 15073 consensus_queue.cc:1046] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250623 01:58:48.049882 15095 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:48.088786 15095 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:48.089277 15095 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:48.051844 15189 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20250623 01:58:48.054527 15040 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.092715 15040 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.094506 15040 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:48.055794 14970 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:48.056448 15107 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:48.096319 15107 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:48.096798 15107 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:48.047607 13809 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250623 01:58:47.998386 15150 raft_consensus.cc:738] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:48.004364 15135 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:48.012425 15132 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:48.072351 13962 rpcz_store.cc:276] 0623 01:58:47.027450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.027560 (+ 110us) service_pool.cc:225] Handling call
0623 01:58:47.302370 (+274810us) spinlock_profiling.cc:243] Waited 9.16 ms on lock 0x7b5800016e98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:58:47.302549 (+ 179us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:47.311065 (+ 8516us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:47.311100 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:47.311576 (+ 476us) log.cc:844] Serialized 26 byte log entry
0623 01:58:47.848461 (+536885us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:47.848517 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:47.848611 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:48.071519 (+222908us) raft_consensus.cc:1683] finished
0623 01:58:48.071546 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:48.071789 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":417,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":19588,"spinlock_wait_cycles":21065088,"thread_start_us":476978,"threads_started":1,"wal-append.queue_time_us":737752}
I20250623 01:58:48.079428 13955 rpcz_store.cc:276] 0623 01:58:47.025877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.028058 (+ 2181us) service_pool.cc:225] Handling call
0623 01:58:47.028314 (+ 256us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:47.068986 (+ 40672us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:47.069026 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:47.081767 (+ 12741us) log.cc:844] Serialized 26 byte log entry
0623 01:58:47.346428 (+264661us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:47.802183 (+455755us) log.cc:844] Serialized 14 byte log entry
0623 01:58:47.805409 (+ 3226us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:47.805510 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:48.078703 (+273193us) raft_consensus.cc:1683] finished
0623 01:58:48.078722 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:48.078865 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":97544,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":196144,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":26068,"thread_start_us":637513,"threads_started":3,"wal-append.queue_time_us":480702}
I20250623 01:58:48.021876 14964 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:48.098353 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.069375 15097 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.125176 13812 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:58:48.029698 15017 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:48.148491 13809 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250623 01:58:48.148957 13809 pending_rounds.cc:85] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 1
I20250623 01:58:48.149214 13809 pending_rounds.cc:107] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250623 01:58:48.149518 13809 raft_consensus.cc:2887] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:48.132733 14922 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:48.012611 15163 consensus_queue.cc:237] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.164187 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:48.138633 15181 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:48.139264 14886 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:48.106293 15177 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:48.016810 14979 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.021863 15087 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.120s
I20250623 01:58:48.061913 15190 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:48.004683 15169 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.108s
I20250623 01:58:48.099129 15095 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.165010 13811 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:48.176769 15126 consensus_meta.cc:287] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.508s user 0.010s sys 0.000s
W20250623 01:58:48.179610 13805 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.017s sys 0.052s
I20250623 01:58:48.180495 15190 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:48.154248 15150 consensus_queue.cc:260] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.184324 13809 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:48.167810 15040 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
W20250623 01:58:48.196974 13805 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.561s user 0.017s sys 0.052s
I20250623 01:58:48.188808 15126 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393) with higher term. Message: Invalid argument: T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
W20250623 01:58:48.177197 15012 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.001s sys 0.026s
W20250623 01:58:48.207669 13957 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.019s sys 0.009s
I20250623 01:58:48.202147 15190 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:48.198516 15054 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.003s sys 0.047s
W20250623 01:58:48.245819 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 488567us
I20250623 01:58:48.247512 13926 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:48.249341 13926 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:48.198253 15177 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.233778 15150 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:48.230597 15012 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.001s sys 0.027s
I20250623 01:58:48.315721 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:48.212340 15040 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:48.232599 13957 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s user 0.019s sys 0.010s
W20250623 01:58:48.296675 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 47457.7us
I20250623 01:58:48.207280 14886 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.320251 13962 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:48.271968 15054 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s user 0.003s sys 0.048s
I20250623 01:58:48.395041 15111 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:48.349462 15093 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:48.272590 15178 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250623 01:58:48.308023 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 66386us
I20250623 01:58:48.306836 15132 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:48.582878 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:48.317299 14764 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:48.317212 14922 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:48.579950 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.596354 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:48.599071 13961 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:48.461540 15097 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:48.344741 14967 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.022s sys 0.011s
W20250623 01:58:48.348881 13777 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.005s sys 0.030s
W20250623 01:58:48.421430 15203 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.188s user 0.000s sys 0.000s
I20250623 01:58:48.349190 15126 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
W20250623 01:58:48.458782 15125 consensus_meta.cc:287] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.664s user 0.008s sys 0.000s
W20250623 01:58:48.398362 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:48.274312 15045 consensus_meta.cc:287] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.788s user 0.011s sys 0.000s
I20250623 01:58:48.423487 15095 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.483692 15177 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:48.490980 15038 consensus_meta.cc:287] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.505s user 0.005s sys 0.004s
I20250623 01:58:48.491508 14979 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.493227 15153 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
W20250623 01:58:48.536248 13810 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.007s sys 0.018s
W20250623 01:58:48.643335 13810 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s user 0.007s sys 0.018s
I20250623 01:58:48.590960 15176 consensus_queue.cc:1035] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
W20250623 01:58:48.396765 14882 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.018s sys 0.046s
I20250623 01:58:48.453012 15107 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:48.461532 13927 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.018s sys 0.031s
W20250623 01:58:48.646783 13927 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.022s sys 0.033s
I20250623 01:58:48.459599 14964 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:48.483101 15131 consensus_meta.cc:287] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.928s user 0.005s sys 0.005s
I20250623 01:58:48.523001 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 60) took 2556 ms. Trace:
W20250623 01:58:48.639463 15112 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.011s sys 0.012s
W20250623 01:58:48.521279 15152 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.006s sys 0.016s
W20250623 01:58:48.649513 14882 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.848s user 0.037s sys 0.061s
W20250623 01:58:48.542320 15065 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.033s sys 0.032s
I20250623 01:58:48.581823 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.674137 13959 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:48.675227 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:48.676470 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.583475 13954 raft_consensus.cc:1238] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:48.594259 13962 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:48.678248 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.678961 13962 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:48.595032 13956 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:48.680714 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.681293 13956 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:48.682301 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:48.521503 13808 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.010s sys 0.016s
W20250623 01:58:48.577605 14514 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.027s sys 0.067s
W20250623 01:58:48.600836 13777 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.008s sys 0.036s
I20250623 01:58:48.567531 15017 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:48.604995 13955 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:48.580783 15178 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:58:48.609001 13776 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.011s sys 0.014s
I20250623 01:58:48.578984 15209 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.595872 15040 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:48.577229 15034 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
W20250623 01:58:48.603139 14967 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.031s sys 0.013s
I20250623 01:58:48.598361 14922 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:48.575866 15181 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20250623 01:58:48.600474 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:48.388732 13960 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.030s sys 0.054s
W20250623 01:58:48.721100 13960 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.931s user 0.030s sys 0.055s
I20250623 01:58:48.644127 13810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1136) took 1286 ms. Trace:
I20250623 01:58:48.721997 13779 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:48.581223 15093 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.566155 13812 raft_consensus.cc:604] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:48.601132 14764 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:48.564922 13925 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.008s sys 0.016s
I20250623 01:58:48.725819 13812 raft_consensus.cc:612] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250623 01:58:48.725934 13925 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.750s user 0.008s sys 0.016s
I20250623 01:58:48.722052 13810 rpcz_store.cc:276] 0623 01:58:47.357795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.357948 (+ 153us) service_pool.cc:225] Handling call
0623 01:58:47.358200 (+ 252us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:47.358367 (+ 167us) raft_consensus.cc:1532] Early marking committed up to index 2
0623 01:58:47.955425 (+597058us) log.cc:844] Serialized 14 byte log entry
0623 01:58:48.643769 (+688344us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:48.643831 (+ 62us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:48.643885 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:48.643948 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:48.644075 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":436851,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":68,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":17779,"thread_start_us":1284692,"threads_started":2}
I20250623 01:58:48.598030 15132 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.580610 15111 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:48.665099 15112 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.580s user 0.012s sys 0.012s
I20250623 01:58:48.730676 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.666563 13709 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:48.684545 13854 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:48.732024 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 47594.4us
W20250623 01:58:48.683634 13808 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s user 0.010s sys 0.016s
I20250623 01:58:48.679904 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:48.693992 13956 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250623 01:58:48.698318 13776 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.011s sys 0.014s
W20250623 01:58:48.734382 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 730050us
I20250623 01:58:48.734784 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.632484 15126 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:48.668527 15152 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.696s user 0.006s sys 0.017s
W20250623 01:58:48.694648 14514 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.028s sys 0.067s
W20250623 01:58:48.673688 15065 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.034s user 0.033s sys 0.032s
W20250623 01:58:48.707777 13852 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.013s sys 0.057s
I20250623 01:58:48.680995 15095 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:48.796526 13852 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.753s user 0.014s sys 0.057s
W20250623 01:58:48.796900 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 754090us
I20250623 01:58:48.646992 15153 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.652598 13708 rpcz_store.cc:276] 0623 01:58:45.966206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:45.966449 (+ 243us) service_pool.cc:225] Handling call
0623 01:58:48.522929 (+2556480us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:46.944365 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:46.945104 (+ 739us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:46.945150 (+ 46us) write_op.cc:435] Acquired schema lock
0623 01:58:46.945175 (+ 25us) tablet.cc:660] Decoding operations
0623 01:58:46.946067 (+ 892us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:46.946156 (+ 89us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:46.946196 (+ 40us) tablet.cc:683] Acquiring locks for 5 operations
0623 01:58:46.946873 (+ 677us) tablet.cc:699] Row locks acquired
0623 01:58:46.946894 (+ 21us) write_op.cc:260] PREPARE: finished
0623 01:58:46.947080 (+ 186us) write_op.cc:270] Start()
0623 01:58:46.947214 (+ 134us) write_op.cc:276] Timestamp: P: 1750643926947009 usec, L: 0
0623 01:58:46.947346 (+ 132us) op_driver.cc:348] REPLICATION: starting
0623 01:58:46.948160 (+ 814us) log.cc:844] Serialized 2099 byte log entry
0623 01:58:47.152381 (+204221us) op_driver.cc:464] REPLICATION: finished
0623 01:58:47.669737 (+517356us) write_op.cc:301] APPLY: starting
0623 01:58:47.669823 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:47.669979 (+ 156us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:47.744331 (+ 74352us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:47.745163 (+ 832us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:47.745439 (+ 276us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0623 01:58:47.745506 (+ 67us) write_op.cc:312] APPLY: finished
0623 01:58:47.745814 (+ 308us) log.cc:844] Serialized 46 byte log entry
0623 01:58:47.745997 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:47.746100 (+ 103us) write_op.cc:454] Released schema lock
0623 01:58:47.746138 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:47.746487 (+ 349us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:47.746513 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":450295,"apply.run_cpu_time_us":2629,"apply.run_wall_time_us":76726,"num_ops":5,"prepare.queue_time_us":114303,"prepare.run_cpu_time_us":4056,"prepare.run_wall_time_us":11617,"replication_time_us":171309,"thread_start_us":360775,"threads_started":2}]]}
I20250623 01:58:48.698984 15210 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:48.701056 15178 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:48.730588 13779 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:48.735885 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.736449 13709 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:48.803905 13956 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:48.804414 13709 catalog_manager.cc:5582] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
W20250623 01:58:48.806100 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 75600.1us
I20250623 01:58:48.806974 13709 catalog_manager.cc:5582] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
W20250623 01:58:48.762681 15194 log.cc:927] Time spent T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.411s user 0.004s sys 0.000s
I20250623 01:58:48.647863 15125 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.809849 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.810461 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:48.810818 13756 thread.cc:640] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.019s sys 0.037s
W20250623 01:58:48.811164 13756 thread.cc:606] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.659s user 0.019s sys 0.037s
I20250623 01:58:48.811177 14882 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:48.813390 13961 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:48.814275 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.816222 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.816646 13779 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:48.818781 13961 raft_consensus.cc:1238] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:48.817788 13779 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:48.819703 14882 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:48.820396 14882 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:48.821585 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:48.747570 15111 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.729965 14886 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:48.729499 13953 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:48.704792 15209 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.722399 15131 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:48.705232 15017 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.720819 15177 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:48.709179 15040 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.785878 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:48.756745 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:48.801782 13805 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.017s sys 0.035s
I20250623 01:58:48.748551 13954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1048) took 1212 ms. Trace:
I20250623 01:58:48.873917 13954 rpcz_store.cc:276] 0623 01:58:47.535944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.537124 (+ 1180us) service_pool.cc:225] Handling call
0623 01:58:47.537492 (+ 368us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:48.677483 (+1139991us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:48.748235 (+ 70752us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400337a54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:48.748462 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2405061888}
I20250623 01:58:48.876044 15215 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:48.823208 15065 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:48.751926 15213 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20250623 01:58:48.792366 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:48.885382 15126 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:48.762089 14764 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.771778 13807 raft_consensus.cc:2391] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:48.772125 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1150) took 1397 ms. Trace:
I20250623 01:58:48.887023 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.886441 13808 rpcz_store.cc:276] 0623 01:58:47.374585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.375515 (+ 930us) service_pool.cc:225] Handling call
0623 01:58:47.440690 (+ 65175us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:47.440870 (+ 180us) raft_consensus.cc:1532] Early marking committed up to index 2
0623 01:58:47.944500 (+503630us) log.cc:844] Serialized 14 byte log entry
0623 01:58:48.771820 (+827320us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:48.771869 (+ 49us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:48.771908 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:48.771980 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:48.772077 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38576,"prepare.queue_time_us":373150,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":38703,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2804,"spinlock_wait_cycles":7296,"thread_start_us":1276707,"threads_started":2,"wal-append.queue_time_us":814720}
I20250623 01:58:48.887571 13807 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
W20250623 01:58:48.889789 15206 log.cc:927] Time spent T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.165s user 0.000s sys 0.000s
I20250623 01:58:48.890821 15176 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:48.891445 15176 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.894531 15082 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:48.895068 15017 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:48.896613 15153 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:48.898499 15209 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:48.900653 14967 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:48.720619 14922 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:48.901122 14967 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.867187 15221 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:48.902123 13812 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:58:48.902359 15221 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:48.749720 15207 log.cc:927] Time spent T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
I20250623 01:58:48.903048 14967 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:48.796044 14478 thread.cc:640] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.043s sys 0.111s
W20250623 01:58:48.904209 14478 thread.cc:606] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.849s user 0.044s sys 0.111s
I20250623 01:58:48.904414 15221 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:48.908252 15125 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.679857 14927 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:48.755684 14964 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:48.748416 15214 log.cc:927] Time spent T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
W20250623 01:58:48.747893 15212 log.cc:927] Time spent T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
I20250623 01:58:48.833081 15038 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
W20250623 01:58:48.748924 15211 log.cc:927] Time spent T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20250623 01:58:48.744724 15132 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:48.738117 15163 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.017s sys 0.026s
I20250623 01:58:48.756521 14979 raft_consensus.cc:2802] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:48.769552 15117 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:48.737632 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:48.917088 13779 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:48.917486 13882 raft_consensus.cc:2408] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250623 01:58:48.742856 15093 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.825126 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.820861 13961 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250623 01:58:48.793262 14970 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:48.833684 13709 catalog_manager.cc:5582] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:48.918891 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.862429 13956 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:48.920300 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:48.920811 13956 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 5
I20250623 01:58:48.797775 15106 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:48.863894 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1047) took 1328 ms. Trace:
I20250623 01:58:48.865407 13806 raft_consensus.cc:2366] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:48.820730 14882 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.758962 15190 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.876502 15215 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:48.876634 15219 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.860780 15135 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:48.869731 15131 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.876894 15177 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.804332 15210 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.881848 14926 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:48.877089 14886 consensus_queue.cc:1046] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20250623 01:58:48.871513 13884 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:48.802947 15095 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.833220 15097 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20250623 01:58:48.846943 15085 consensus_queue.cc:1046] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20250623 01:58:48.831104 15150 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250623 01:58:48.831604 15054 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.950495 15132 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.882959 15073 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:48.831276 15087 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20250623 01:58:48.961614 14967 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:48.884761 13811 consensus_meta.cc:287] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.678s user 0.007s sys 0.003s
I20250623 01:58:48.902719 13812 raft_consensus.cc:604] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:48.864467 15012 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.778512 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:48.773691 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:48.878754 13805 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.017s sys 0.043s
I20250623 01:58:48.923612 13813 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:48.998857 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.940054 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:48.942140 13779 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:48.913820 15065 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.996779 13811 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:48.945955 15106 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.938244 13958 rpcz_store.cc:276] 0623 01:58:47.535651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.535786 (+ 135us) service_pool.cc:225] Handling call
0623 01:58:47.536142 (+ 356us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:47.536486 (+ 344us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:47.536523 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:47.537011 (+ 488us) log.cc:844] Serialized 26 byte log entry
0623 01:58:47.883305 (+346294us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:47.931191 (+ 47886us) log.cc:844] Serialized 14 byte log entry
0623 01:58:47.931411 (+ 220us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:47.931480 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:48.083674 (+152194us) spinlock_profiling.cc:243] Waited 7.58 ms on lock 0x7b180033ac00. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b404f2 00007f41e0b428c1 00007f41e0b42879 00007f41e0b42809 00007f41e0b427d1 00007f41e0b41acd 00007f41e9b9b124 00007f41e9b9af59 00007f41e0b23646
0623 01:58:48.833400 (+749726us) raft_consensus.cc:1683] finished
0623 01:58:48.863678 (+ 30278us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:48.863834 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":47614,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":26193,"raft.queue_time_us":74337,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":2674,"spinlock_wait_cycles":17434880,"thread_start_us":341466,"threads_started":1,"wal-append.queue_time_us":647895}
I20250623 01:58:48.937194 13709 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:48.949086 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:48.956460 15221 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.025498 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.026072 13958 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:49.026854 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.027752 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:49.028195 13958 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:48.952291 13926 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:58:48.953039 13806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 530) took 1419 ms (client timeout 1820 ms). Trace:
W20250623 01:58:49.031044 13806 rpcz_store.cc:269] 0623 01:58:47.534038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.536169 (+ 2131us) service_pool.cc:225] Handling call
0623 01:58:48.952766 (+1416597us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54001fa9d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:48.952969 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2999187328}
I20250623 01:58:48.919970 14979 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:48.936309 15126 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.867022 15107 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:48.964074 13956 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:58:48.958519 15219 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.936655 15117 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:48.912053 15040 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.939549 15176 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:48.948244 15017 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:48.882717 15045 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:48.945822 14882 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:48.883991 15169 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:48.977715 13960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1136) took 1227 ms. Trace:
I20250623 01:58:49.038013 14979 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:48.945549 15125 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250623 01:58:48.944098 14970 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.976024 15085 raft_consensus.cc:3053] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:49.016659 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 526) took 1546 ms. Trace:
I20250623 01:58:49.118954 13811 rpcz_store.cc:276] 0623 01:58:47.470268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.482759 (+ 12491us) service_pool.cc:225] Handling call
0623 01:58:47.673683 (+190924us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:48.163998 (+490315us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b031ec 00007f41e0b02dab 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:49.016587 (+852589us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8588416}
I20250623 01:58:49.050406 15181 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.958937 15135 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:48.916514 14927 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.016000 14886 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:48.976714 15054 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:48.977440 15111 consensus_queue.cc:1046] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20250623 01:58:48.979197 15132 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.064579 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.082535 13958 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:58:49.018766 15222 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250623 01:58:49.090634 13960 rpcz_store.cc:276] 0623 01:58:47.749769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.749871 (+ 102us) service_pool.cc:225] Handling call
0623 01:58:47.750173 (+ 302us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:47.753008 (+ 2835us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:47.753036 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:47.753437 (+ 401us) log.cc:844] Serialized 26 byte log entry
0623 01:58:48.053631 (+300194us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b4800211a58. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e8956871 00007f41e894da17 00007f41e894d8aa 00007f41e8a4297f 00007f41e8a42647 00007f41e8a56c6c 00007f41e8a56c18 00007f41e8a56b98 00007f41e8a56b53 00007f41e8a55dfc 00007f41e79dc736
0623 01:58:48.721690 (+668059us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:48.721753 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:48.721846 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:48.974521 (+252675us) raft_consensus.cc:1683] finished
0623 01:58:48.974546 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:48.977632 (+ 3086us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2469,"raft.queue_time_us":5006,"raft.run_cpu_time_us":2165,"raft.run_wall_time_us":286461,"spinlock_wait_cycles":16433920,"thread_start_us":931705,"threads_started":1,"wal-append.queue_time_us":820161}
I20250623 01:58:48.968129 15093 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:48.997232 14967 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:48.915149 15163 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s user 0.017s sys 0.027s
I20250623 01:58:49.061787 15117 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:49.089339 14922 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.137372 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:48562 (request call id 479) took 1048 ms. Trace:
I20250623 01:58:49.064283 13804 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20250623 01:58:49.140695 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 269710us
I20250623 01:58:49.098138 15017 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:49.036511 15221 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.098600 15176 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:49.024502 15065 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.099860 15087 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:49.146428 13956 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: []
W20250623 01:58:49.132637 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 237118us
I20250623 01:58:49.051185 15126 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.189452 15190 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.191923 15017 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:49.164170 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.165550 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.166066 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.167683 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.169484 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.131955 15085 raft_consensus.cc:738] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:49.223774 15085 consensus_queue.cc:260] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.180991 15229 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:49.137085 15034 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.019040 15012 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.225448 15107 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.226071 15219 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:49.226011 15107 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.226903 15012 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.109509 14764 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250623 01:58:49.135403 15209 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.142805 15038 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:49.183857 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.184969 13809 rpcz_store.cc:276] 0623 01:58:48.073718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:48.206462 (+132744us) service_pool.cc:225] Handling call
0623 01:58:49.121580 (+915118us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b540016c1d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa5315 00007f41e3a9d5ea 00007f41e89f1b09 00007f41e208f244 00007f41e208f1d2 00007f41e208f101 00007f41e208f07c 00007f41e208e332 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:49.121798 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1864479616}
I20250623 01:58:49.121026 15169 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:49.244406 15038 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:49.244020 15117 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.227838 15107 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.245862 13887 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:49.246361 15038 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:49.248095 15117 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.248301 14967 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:49.248694 13957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1049) took 1712 ms. Trace:
I20250623 01:58:49.143757 15181 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:49.192698 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 250622us
W20250623 01:58:49.197888 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 181515us
I20250623 01:58:49.256840 13779 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:49.185885 15131 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.279434 15135 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:49.279505 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 272c03e6252d489cbf2fdceb36b05dc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:49.175316 15230 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:49.161058 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.280439 15230 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:49.280833 15230 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:49.281414 14927 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.282042 15126 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.283082 15132 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.280853 15221 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:49.228363 13925 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 1.820s (SENT) [suppressed 73 similar messages]
I20250623 01:58:49.286363 15242 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.196954 15176 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.209761 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:58:49.215044 15227 log.cc:927] Time spent T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20250623 01:58:49.215222 15095 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:49.215306 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.215638 15210 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.216364 15153 raft_consensus.cc:3053] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:49.182538 15093 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.145541 14886 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:49.202500 15215 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:49.203488 15177 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:49.230559 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:49.246357 15169 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:49.210000 15087 raft_consensus.cc:695] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:49.233711 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.234239 15054 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:49.225329 15229 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.293035 15228 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.239522 15106 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.240674 14922 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:49.241545 15209 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.242180 15112 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:49.242548 14970 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.242091 15178 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.243106 15040 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:49.251432 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:49.334131 13956 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:49.254177 13853 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:49.258797 13776 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:49.337950 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 79212.9us
I20250623 01:58:49.339141 13776 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:49.278683 15240 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:49.278591 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.287600 13809 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:58:49.256568 14967 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.287364 15163 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.225391 15034 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.289201 15219 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.313261 13882 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
W20250623 01:58:49.318917 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 366690us
I20250623 01:58:49.325873 15054 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:49.317848 15215 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.295676 15132 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.328438 15177 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.246403 14882 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.298576 15153 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:49.350045 15240 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:49.341102 13809 raft_consensus.cc:604] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:49.295913 15097 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
W20250623 01:58:49.350594 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 96517.6us
I20250623 01:58:49.359123 13956 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:49.316560 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.220939 15150 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:49.378926 13811 pending_rounds.cc:85] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 1
I20250623 01:58:49.379590 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:49.382788 14478 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:49.326491 13810 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:49.387499 13962 raft_consensus.cc:2366] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:58:49.286871 15242 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.362102 15126 consensus_queue.cc:1046] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.107s
I20250623 01:58:49.348943 15112 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.330296 15228 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.389015 13926 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:58:49.389101 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 201120us
I20250623 01:58:49.383423 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1151) took 2007 ms. Trace:
I20250623 01:58:49.346680 15241 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:49.349733 15125 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.349263 14922 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.669437 13809 raft_consensus.cc:612] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250623 01:58:49.498587 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:49.527962 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.361666 15107 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:49.249095 13957 rpcz_store.cc:276] 0623 01:58:47.536239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.539223 (+ 2984us) service_pool.cc:225] Handling call
0623 01:58:47.592562 (+ 53339us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:47.645556 (+ 52994us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:47.645607 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:47.646106 (+ 499us) log.cc:844] Serialized 26 byte log entry
0623 01:58:48.600258 (+954152us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:49.065041 (+464783us) log.cc:844] Serialized 14 byte log entry
0623 01:58:49.134781 (+ 69740us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.248311 (+113530us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b540021d254. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:49.248360 (+ 49us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:49.248405 (+ 45us) raft_consensus.cc:1683] finished
0623 01:58:49.248510 (+ 105us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:49.248639 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23963,"prepare.queue_time_us":580741,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":3112,"raft.queue_time_us":100546,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":2373,"spinlock_wait_cycles":12614912,"thread_start_us":1244274,"threads_started":2,"wal-append.queue_time_us":854131}
W20250623 01:58:49.786417 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 91530.8us
W20250623 01:58:49.786221 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 96287.4us
I20250623 01:58:49.561364 15178 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:49.681744 13954 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:49.509511 15087 consensus_queue.cc:237] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.804574 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.388687 15215 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.807040 13811 pending_rounds.cc:107] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250623 01:58:49.816406 13811 raft_consensus.cc:2887] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250623 01:58:49.466640 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 86378.3us
I20250623 01:58:49.810472 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.470125 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:49.521566 15229 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:49.823987 13925 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.007s sys 0.006s
I20250623 01:58:49.360343 15163 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.384702 15135 consensus_queue.cc:1046] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.481s
I20250623 01:58:49.936228 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:49.530043 15209 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:49.937351 13925 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.650s user 0.008s sys 0.006s
W20250623 01:58:49.938009 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 709627us
I20250623 01:58:49.841805 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:49.853873 13779 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:49.854703 13956 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:49.552809 15246 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:49.727722 15240 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:49.390035 15054 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.383939 15040 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:49.522168 15231 log.cc:927] Time spent T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
W20250623 01:58:49.520126 15239 log.cc:927] Time spent T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20250623 01:58:49.842052 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1156) took 1180 ms. Trace:
I20250623 01:58:49.966938 13955 rpcz_store.cc:276] 0623 01:58:48.661930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.155059 (+493129us) service_pool.cc:225] Handling call
0623 01:58:49.155400 (+ 341us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:49.389454 (+234054us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:49.389487 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:49.389529 (+ 42us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:49.389558 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.841803 (+452245us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54003efdd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:49.841850 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:49.841991 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":537835904}
I20250623 01:58:49.865064 15107 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:49.531997 15226 log.cc:927] Time spent T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.558s user 0.001s sys 0.001s
I20250623 01:58:49.808581 13959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1160) took 1145 ms. Trace:
I20250623 01:58:49.838598 15210 raft_consensus.cc:991] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
W20250623 01:58:49.911293 15225 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.382s user 0.000s sys 0.001s
I20250623 01:58:49.537580 15085 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:49.384490 15065 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:49.931270 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 548551us
I20250623 01:58:49.360502 14967 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.917676 13962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 1163) took 1254 ms. Trace:
W20250623 01:58:49.510250 15245 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20250623 01:58:49.384668 14970 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:49.524590 15093 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.027328 15065 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:50.027819 15093 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:50.028007 15065 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.028359 15093 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.958768 15246 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.030313 15065 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:49.386520 15219 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.520208 14886 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:49.537721 15250 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.947686 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1159) took 1285 ms. Trace:
I20250623 01:58:50.048179 13961 rpcz_store.cc:276] 0623 01:58:48.662582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.150076 (+487494us) service_pool.cc:225] Handling call
0623 01:58:49.150418 (+ 342us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:49.150671 (+ 253us) raft_consensus.cc:1532] Early marking committed up to index 3
0623 01:58:49.387702 (+237031us) log.cc:844] Serialized 14 byte log entry
0623 01:58:49.946844 (+559142us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:49.947058 (+ 214us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:49.947188 (+ 130us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.947340 (+ 152us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:49.947594 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":67617,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":778873,"threads_started":3,"wal-append.queue_time_us":640486}
I20250623 01:58:49.417207 15243 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:49.513739 15131 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:49.941416 13809 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
W20250623 01:58:49.959262 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 102306us
I20250623 01:58:50.055758 15065 raft_consensus.cc:1079] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:58:49.665566 13805 rpcz_store.cc:276] 0623 01:58:47.375803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:47.410358 (+ 34555us) service_pool.cc:225] Handling call
0623 01:58:47.634790 (+224432us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b0323e 0000000000356d09 00007f41e897bdc0 00007f41e8a427d4 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:58:47.634979 (+ 189us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:47.635188 (+ 209us) raft_consensus.cc:1532] Early marking committed up to index 2
0623 01:58:48.232330 (+597142us) log.cc:844] Serialized 14 byte log entry
0623 01:58:49.308182 (+1075852us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:49.312184 (+ 4002us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:49.323568 (+ 11384us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.330367 (+ 6799us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:49.383352 (+ 52985us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10679,"prepare.queue_time_us":295250,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":186639,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":6237,"spinlock_wait_cycles":4679552,"thread_start_us":1602503,"threads_started":3,"wal-append.queue_time_us":824062}
I20250623 01:58:50.055007 15267 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:50.001963 15153 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.057777 15267 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.058313 15153 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:49.964013 13854 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:50.059293 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 95353.2us
I20250623 01:58:50.059559 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 539) took 1155 ms. Trace:
I20250623 01:58:50.060463 13854 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:50.063951 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.064918 13883 raft_consensus.cc:2391] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:49.965385 13813 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
W20250623 01:58:49.806748 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 337995us
I20250623 01:58:49.968535 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.865192 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:49.557876 15017 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.417572 15177 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.785544 14922 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.786827 15125 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.806506 15150 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.992981 13959 rpcz_store.cc:276] 0623 01:58:48.662802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.180509 (+517707us) service_pool.cc:225] Handling call
0623 01:58:49.180812 (+ 303us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:49.375239 (+194427us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:49.375282 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:49.375335 (+ 53us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:49.375374 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.808319 (+432945us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400073854. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:49.808378 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:49.808521 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":446589696}
I20250623 01:58:50.087159 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:49.978703 15040 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.941049 15178 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
W20250623 01:58:50.002169 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 613216us
I20250623 01:58:50.102360 13926 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:50.008569 13962 rpcz_store.cc:276] 0623 01:58:48.663411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:48.679665 (+ 16254us) service_pool.cc:225] Handling call
0623 01:58:49.917433 (+1237768us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b5400244ad4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:49.917601 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1502478592}
I20250623 01:58:50.104718 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.105106 13854 leader_election.cc:400] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:58:50.105454 13962 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:49.555542 15230 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.958410 15209 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:50.006754 15190 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:50.117476 15150 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.117764 15190 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.118163 15209 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:50.118642 15209 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:49.392192 15221 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:50.119664 15221 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.120043 15267 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.120496 15274 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:50.120980 15274 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:50.121528 15221 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.122112 13810 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:49.957679 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.936373 13954 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:50.031512 15135 consensus_queue.cc:1230] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250623 01:58:49.523025 14979 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:50.122934 15274 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:49.933770 15229 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:50.123694 15229 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:50.124948 14979 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:50.126233 15190 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.126530 15230 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.126839 15229 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:50.055095 15126 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:50.131634 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.066056 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.059980 13807 rpcz_store.cc:276] 0623 01:58:48.904585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:48.910583 (+ 5998us) service_pool.cc:225] Handling call
0623 01:58:48.910888 (+ 305us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:48.940709 (+ 29821us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:48.940748 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:48.941343 (+ 595us) log.cc:844] Serialized 26 byte log entry
0623 01:58:49.298986 (+357643us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:49.387330 (+ 88344us) log.cc:844] Serialized 14 byte log entry
0623 01:58:49.511858 (+124528us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.511966 (+ 108us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:50.059333 (+547367us) raft_consensus.cc:1683] finished
0623 01:58:50.059370 (+ 37us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:50.059503 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":110871,"prepare.queue_time_us":223725,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":337568,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":157653,"thread_start_us":318156,"threads_started":2,"wal-append.queue_time_us":386529}
I20250623 01:58:49.786705 15241 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.070144 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.136391 13880 raft_consensus.cc:2408] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:50.070789 13955 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:50.137607 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.138302 13955 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:50.073853 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.139680 13886 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:50.078477 13883 raft_consensus.cc:1273] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250623 01:58:50.085960 13882 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250623 01:58:50.095048 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:49.468786 15132 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.088222 15038 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:50.097633 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:50.105474 13854 leader_election.cc:403] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250623 01:58:50.001410 15107 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.115629 15125 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.150907 13854 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:50.151818 13854 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:50.152616 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.124313 15274 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:50.124850 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:49.966850 15198 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250623 01:58:50.032857 15234 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.011s sys 0.013s
I20250623 01:58:50.032450 14967 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.075795 14886 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:50.131304 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.193992 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:50.194689 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 340874us
I20250623 01:58:50.153383 13955 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:50.195789 13779 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:50.196444 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.197190 13955 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
W20250623 01:58:50.197597 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:50.198348 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.151856 13887 raft_consensus.cc:2391] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:50.157560 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.110484 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.177217 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.201104 13885 raft_consensus.cc:2408] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:50.180508 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.181115 15097 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:50.182539 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.185547 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:50.202623 13886 raft_consensus.cc:2391] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:50.188632 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.135825 15126 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:50.197206 13779 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:50.209962 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.195128 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.205932 13854 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:50.195127 13961 raft_consensus.cc:2408] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250623 01:58:50.168928 15234 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s user 0.011s sys 0.014s
I20250623 01:58:50.151639 15038 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:50.225626 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.227366 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.211738 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:50.210580 13882 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:50.146989 15242 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.211514 13779 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:50.055073 15243 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.234259 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.234295 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.237712 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:50.242856 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.245710 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.169489 15198 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:50.071240 15250 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.254846 13812 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:50.255383 15242 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:50.256870 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 543) took 1343 ms. Trace:
I20250623 01:58:50.290575 13808 rpcz_store.cc:276] 0623 01:58:48.913960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.000101 (+ 86141us) service_pool.cc:225] Handling call
0623 01:58:49.032912 (+ 32811us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:49.326023 (+293111us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:49.326055 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:49.333274 (+ 7219us) log.cc:844] Serialized 26 byte log entry
0623 01:58:49.823743 (+490469us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:49.835091 (+ 11348us) log.cc:844] Serialized 14 byte log entry
0623 01:58:49.835318 (+ 227us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.835405 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:50.256653 (+421248us) raft_consensus.cc:1683] finished
0623 01:58:50.256675 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:50.256813 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":899,"prepare.run_cpu_time_us":8843,"prepare.run_wall_time_us":39120,"raft.queue_time_us":339806,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":31176,"thread_start_us":648625,"threads_started":2,"wal-append.queue_time_us":783362}
I20250623 01:58:50.258141 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:50.292549 13880 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:50.259680 15243 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:50.296567 15249 log.cc:927] Time spent T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20250623 01:58:50.204341 14882 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.262338 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1165) took 1598 ms. Trace:
I20250623 01:58:50.218457 15125 raft_consensus.cc:2747] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:50.297712 13961 rpcz_store.cc:276] 0623 01:58:48.663827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.232488 (+1568661us) service_pool.cc:225] Handling call
0623 01:58:50.261731 (+ 29243us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:50.261945 (+ 214us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:50.261983 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:50.262038 (+ 55us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:50.262080 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:50.262149 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:50.262262 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:50.207849 15228 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.265035 15150 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:50.266228 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:50.266433 15241 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.267306 13960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1158) took 1604 ms. Trace:
I20250623 01:58:50.242671 15251 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:50.218815 15126 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:50.270804 14507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48412 (request call id 67) took 1027 ms. Trace:
I20250623 01:58:50.129508 14479 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:50.132427 15178 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:50.249743 15271 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:50.271168 15177 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.216427 15270 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:50.236949 15038 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:50.236232 14948 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:50.657972 13813 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7560a94e265b433dbc427ef13c120669"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:58:50.660986 13813 raft_consensus.cc:604] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
W20250623 01:58:50.393715 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 331820us
I20250623 01:58:50.413574 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1166) took 1749 ms. Trace:
I20250623 01:58:50.302217 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:50.324577 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 220917us
I20250623 01:58:50.661990 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:50.675148 13926 leader_election.cc:400] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:50.675218 13813 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:50.675519 13926 leader_election.cc:403] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:50.676543 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:50.671569 13960 rpcz_store.cc:276] 0623 01:58:48.662362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.174573 (+512211us) service_pool.cc:225] Handling call
0623 01:58:49.184984 (+ 10411us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:49.359736 (+174752us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:49.359785 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:49.466143 (+106358us) log.cc:844] Serialized 26 byte log entry
0623 01:58:49.824454 (+358311us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:49.824505 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.824574 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:50.267115 (+442541us) raft_consensus.cc:1683] finished
0623 01:58:50.267134 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:50.267250 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4757,"raft.queue_time_us":31389,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"thread_start_us":463350,"threads_started":2,"wal-append.queue_time_us":743576}
I20250623 01:58:50.678237 13813 raft_consensus.cc:1238] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: []
W20250623 01:58:50.673051 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 397862us
I20250623 01:58:50.664237 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.664821 13961 rpcz_store.cc:276] 0623 01:58:48.664041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.342903 (+1678862us) service_pool.cc:225] Handling call
0623 01:58:50.412871 (+ 69968us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:50.413105 (+ 234us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:50.413148 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:50.413207 (+ 59us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:50.413250 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:50.413341 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:50.413497 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:50.274336 15198 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:50.251768 14926 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:50.704893 13880 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:58:50.665822 13807 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
W20250623 01:58:50.690065 15266 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250623 01:58:50.480249 15277 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:50.667658 15150 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.711195 15244 raft_consensus.cc:3053] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:50.711118 15277 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.711658 15244 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:50.592391 15181 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.301329 15163 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:50.695254 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 606597us
I20250623 01:58:50.381624 15135 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:50.713052 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:50.713452 15135 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.706413 15034 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.686295 14948 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:50.716152 15190 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:50.657963 15281 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:50.716428 15150 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.717126 15281 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.717628 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:50.718284 13956 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:50.718747 13956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 1181) took 1467 ms. Trace:
I20250623 01:58:50.719233 15281 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.720435 15176 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.685078 15246 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.720963 15198 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:50.712481 15244 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:50.696090 15178 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:50.335048 15242 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
W20250623 01:58:50.619707 15153 consensus_meta.cc:287] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.561s user 0.007s sys 0.002s
I20250623 01:58:50.588099 15280 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:50.531898 15209 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.662086 15017 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.684461 15065 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:50.316663 15221 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:50.661198 15252 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:50.315825 15243 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:50.697650 15272 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:50.697712 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:50.698505 15107 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.700030 13962 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
W20250623 01:58:50.700189 15240 consensus_meta.cc:287] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.734s user 0.008s sys 0.000s
I20250623 01:58:50.700665 15149 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:50.701612 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.703208 13810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 551) took 1323 ms. Trace:
I20250623 01:58:50.703285 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 547) took 1325 ms. Trace:
I20250623 01:58:50.704257 13926 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:50.705572 13960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1167) took 2035 ms. Trace:
I20250623 01:58:50.706231 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.697814 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:50.710392 15273 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:50.723330 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:50.695963 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:50.347808 15215 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.723932 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:50.719060 13956 rpcz_store.cc:276] 0623 01:58:49.250996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.950481 (+699485us) service_pool.cc:225] Handling call
0623 01:58:50.717476 (+766995us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b5800016e98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:50.718691 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16027520}
I20250623 01:58:50.722100 14512 raft_consensus.cc:1238] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:50.704007 15040 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.727614 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 540) took 1814 ms. Trace:
I20250623 01:58:50.375008 15054 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.667407 15219 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.442389 13957 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:50.716828 15132 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.024322 15252 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:51.028532 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:51.029462 14514 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:51.029778 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.031566 15126 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:50.827134 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.035537 13956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1169) took 2301 ms. Trace:
I20250623 01:58:51.040951 15198 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.858887 13776 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
W20250623 01:58:50.963688 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 206735us
I20250623 01:58:51.035882 13883 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:51.043195 15252 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:50.934573 15251 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:51.047027 15149 raft_consensus.cc:695] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:50.866178 15038 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:50.965440 13809 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.007s sys 0.009s
I20250623 01:58:51.041509 13807 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
W20250623 01:58:51.085330 13809 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.007s sys 0.009s
I20250623 01:58:51.042658 13810 rpcz_store.cc:276] 0623 01:58:49.379411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.931082 (+551671us) service_pool.cc:225] Handling call
0623 01:58:50.703028 (+771946us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54001dd154. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:50.703138 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":426064128}
W20250623 01:58:50.805111 14886 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.018s sys 0.026s
I20250623 01:58:50.705328 14926 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:51.041270 13960 rpcz_store.cc:276] 0623 01:58:48.670666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.704691 (+2034025us) service_pool.cc:225] Handling call
0623 01:58:50.705001 (+ 310us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:50.705180 (+ 179us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:50.705208 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:50.705248 (+ 40us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:50.705278 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:50.705397 (+ 119us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:50.705512 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:51.047452 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:50.720237 14948 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:50.812649 13887 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.010s sys 0.014s
W20250623 01:58:51.090456 13887 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.834s user 0.010s sys 0.017s
W20250623 01:58:50.723213 15085 consensus_meta.cc:287] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.708s user 0.002s sys 0.007s
I20250623 01:58:51.043305 13811 rpcz_store.cc:276] 0623 01:58:49.378067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.378237 (+ 170us) service_pool.cc:225] Handling call
0623 01:58:49.378658 (+ 421us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:49.854663 (+476005us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:49.854711 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:49.855217 (+ 506us) log.cc:844] Serialized 26 byte log entry
0623 01:58:50.208302 (+353085us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:50.208339 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:50.208400 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:50.703095 (+494695us) raft_consensus.cc:1683] finished
0623 01:58:50.703117 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:50.703232 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":352788,"threads_started":1,"wal-append.queue_time_us":805954}
I20250623 01:58:51.033910 15280 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:50.774199 15244 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:50.349095 15093 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:50.728873 15271 raft_consensus.cc:695] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:51.056900 13853 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:51.064002 13956 rpcz_store.cc:276] 0623 01:58:48.733729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.034438 (+2300709us) service_pool.cc:225] Handling call
0623 01:58:51.034729 (+ 291us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.034939 (+ 210us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.035264 (+ 325us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.035314 (+ 50us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:51.035349 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.035401 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.035494 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:51.131711 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 74958.3us
I20250623 01:58:51.132563 13956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1170) took 2391 ms. Trace:
I20250623 01:58:50.964064 15276 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:50.717000 15135 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:51.063689 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:50.794486 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.044211 15209 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:51.066769 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.135110 13886 raft_consensus.cc:2366] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:50.724858 15277 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:50.665179 14507 rpcz_store.cc:276] 0623 01:58:49.243341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.284806 (+ 41465us) service_pool.cc:225] Handling call
0623 01:58:49.285176 (+ 370us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:49.285431 (+ 255us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:49.285460 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:49.285957 (+ 497us) log.cc:844] Serialized 26 byte log entry
0623 01:58:49.388231 (+102274us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:49.416904 (+ 28673us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.468927 (+ 52023us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:50.270597 (+801670us) raft_consensus.cc:1683] finished
0623 01:58:50.270615 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:50.270746 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":87634,"threads_started":1,"wal-append.queue_time_us":817443}
W20250623 01:58:51.039960 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 335760us
I20250623 01:58:51.082777 15242 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.037684 13804 rpcz_store.cc:276] 0623 01:58:48.913126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:48.915912 (+ 2786us) service_pool.cc:225] Handling call
0623 01:58:49.020507 (+104595us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:49.187379 (+166872us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.327354 (+139975us) spinlock_profiling.cc:243] Waited 9.45 ms on lock 0x7b540016c1d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:50.727395 (+1400041us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b540016c1d0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:50.727551 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128540416}
I20250623 01:58:51.053211 15284 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:51.086282 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:50.716712 15190 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.153362 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.153133 15284 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:51.154603 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.154995 15190 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.038787 15221 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.155561 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.155652 15284 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:51.156673 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:51.156850 15221 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:50.812058 13959 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.014s sys 0.008s
W20250623 01:58:51.157441 13959 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s user 0.016s sys 0.008s
W20250623 01:58:51.084586 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 400081us
I20250623 01:58:51.038621 15243 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.087179 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.038074 15065 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.038203 15272 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.046182 15240 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.159188 13810 raft_consensus.cc:2391] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:51.084106 13953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1152) took 2432 ms. Trace:
I20250623 01:58:51.093123 13883 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:51.079187 15251 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:51.087729 14886 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.846s user 0.027s sys 0.029s
I20250623 01:58:51.084190 14922 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:51.122045 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 538) took 1064 ms. Trace:
I20250623 01:58:51.161451 15272 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:51.123732 15097 consensus_meta.cc:287] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.901s user 0.003s sys 0.005s
I20250623 01:58:51.125769 13879 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:51.127957 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.159605 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.130360 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:51.109851 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:51.132898 13956 rpcz_store.cc:276] 0623 01:58:48.741402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.131854 (+2390452us) service_pool.cc:225] Handling call
0623 01:58:51.132139 (+ 285us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.132294 (+ 155us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.132317 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.132353 (+ 36us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:51.132379 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.132425 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.132517 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:51.091497 14926 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:51.095489 15280 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:51.184983 15240 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.186409 13960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 95) took 2310 ms. Trace:
I20250623 01:58:51.186774 13960 rpcz_store.cc:276] 0623 01:58:48.876332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.185675 (+2309343us) service_pool.cc:225] Handling call
0623 01:58:51.185944 (+ 269us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.186104 (+ 160us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.186132 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.186183 (+ 51us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:58:51.186217 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.186268 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.186359 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:51.180801 15097 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:58:51.149154 13926 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:51.180958 14514 raft_consensus.cc:2408] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250623 01:58:51.138432 13961 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:51.150095 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.190443 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 96) took 2232 ms. Trace:
I20250623 01:58:51.190508 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.190860 13961 rpcz_store.cc:276] 0623 01:58:48.958102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.189707 (+2231605us) service_pool.cc:225] Handling call
0623 01:58:51.189986 (+ 279us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.190156 (+ 170us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.190184 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.190225 (+ 41us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:58:51.190254 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.190305 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.190395 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:51.191609 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.192677 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:51.091325 15273 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20250623 01:58:51.170982 13881 rpcz_store.cc:276] 0623 01:58:50.057520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.069679 (+ 12159us) service_pool.cc:225] Handling call
0623 01:58:51.113516 (+1043837us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:58:51.121475 (+ 7959us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.121667 (+ 192us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.121694 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.121744 (+ 50us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:51.121782 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.121866 (+ 84us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.121986 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42538880}
I20250623 01:58:51.194257 15273 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:51.194491 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.136783 15209 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:51.195173 15209 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:51.195669 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.162537 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.189246 13926 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:51.197666 13810 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:51.198509 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.198894 13926 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:51.199117 13810 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:51.199736 13926 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:51.170230 14886 raft_consensus.cc:695] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:51.192677 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1176) took 2225 ms. Trace:
I20250623 01:58:51.161232 15243 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.174383 13806 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:51.137470 15276 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.201305 14886 consensus_queue.cc:237] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.202216 15209 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.203738 15276 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.204061 15209 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:51.200966 13961 rpcz_store.cc:276] 0623 01:58:48.967146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.191932 (+2224786us) service_pool.cc:225] Handling call
0623 01:58:51.192217 (+ 285us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.192382 (+ 165us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.192408 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.192449 (+ 41us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:51.192481 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.192534 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.192626 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:51.166342 15126 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:51.207377 15045 raft_consensus.cc:3053] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:51.162266 13953 rpcz_store.cc:276] 0623 01:58:48.651939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:48.877773 (+225834us) service_pool.cc:225] Handling call
0623 01:58:48.882107 (+ 4334us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:49.199139 (+317032us) raft_consensus.cc:1532] Early marking committed up to index 3
0623 01:58:49.347081 (+147942us) log.cc:844] Serialized 14 byte log entry
0623 01:58:50.137072 (+789991us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:50.149091 (+ 12019us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:50.158151 (+ 9060us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.083772 (+925621us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54000bd6d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:51.083859 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.084019 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32721,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":295463,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"spinlock_wait_cycles":697988352,"thread_start_us":766502,"threads_started":3,"wal-append.queue_time_us":1267901}
I20250623 01:58:51.207769 15045 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:51.208582 15045 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.129194 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.183470 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.078251 13925 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.215040 13808 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:51.179493 15065 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.135619 15291 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:51.216862 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.161777 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:51.198580 15277 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:51.199745 14922 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:51.210347 13962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 1187) took 1516 ms. Trace:
W20250623 01:58:51.217660 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 55948.7us
I20250623 01:58:51.217775 13962 rpcz_store.cc:276] 0623 01:58:49.694105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.194905 (+500800us) service_pool.cc:225] Handling call
0623 01:58:51.210174 (+1015269us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b54003f7fd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:51.210289 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1120997632}
W20250623 01:58:51.215500 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 137759us
I20250623 01:58:51.194630 15273 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:51.196615 14511 raft_consensus.cc:1238] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:51.213652 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 97) took 2242 ms. Trace:
I20250623 01:58:51.230921 13925 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.230860 13961 rpcz_store.cc:276] 0623 01:58:48.971699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.206221 (+2234522us) service_pool.cc:225] Handling call
0623 01:58:51.213012 (+ 6791us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.213229 (+ 217us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.213270 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.213356 (+ 86us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:58:51.213398 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.213467 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.213589 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:58:51.222157 13954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 1080) took 1835 ms (client timeout 1825 ms). Trace:
I20250623 01:58:51.232314 13852 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
W20250623 01:58:51.232487 13954 rpcz_store.cc:269] 0623 01:58:49.386803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.804230 (+417427us) service_pool.cc:225] Handling call
0623 01:58:51.222000 (+1417770us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b540021be54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:51.222098 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":276175104}
I20250623 01:58:51.232930 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 99) took 2074 ms. Trace:
I20250623 01:58:51.233368 13961 rpcz_store.cc:276] 0623 01:58:49.158889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.231814 (+2072925us) service_pool.cc:225] Handling call
0623 01:58:51.232466 (+ 652us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.232636 (+ 170us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.232659 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.232699 (+ 40us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:58:51.232738 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.232783 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.232883 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:51.224776 13886 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
W20250623 01:58:51.234763 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.235405 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.163519 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.236945 13808 raft_consensus.cc:1238] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250623 01:58:51.121937 15271 consensus_queue.cc:237] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.179456 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.237001 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.240162 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.243710 14478 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.246567 13811 raft_consensus.cc:1238] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 3. Current term is 4. Ops: []
W20250623 01:58:51.247884 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 56934.2us
W20250623 01:58:51.250123 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 272c03e6252d489cbf2fdceb36b05dc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:51.250363 13852 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:51.251276 13811 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:51.252687 13852 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:51.216768 15040 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
W20250623 01:58:51.254014 15295 log.cc:927] Time spent T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20250623 01:58:51.229141 15045 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:51.254495 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.214479 13804 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 1.
I20250623 01:58:51.219030 15287 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:51.216598 15038 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:51.220439 14922 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:51.302033 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 49443.5us
I20250623 01:58:51.302811 13852 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:51.231750 15273 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:51.293390 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:51.299038 13888 raft_consensus.cc:1238] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:58:51.249115 15163 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:58:51.289557 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 550) took 1910 ms (client timeout 1702 ms). Trace:
I20250623 01:58:51.255249 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:58:51.304725 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 73871.5us
I20250623 01:58:51.275228 15073 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:51.275279 15277 raft_consensus.cc:695] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:51.319263 13811 raft_consensus.cc:1238] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:51.229283 15235 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:51.255060 15219 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:51.255800 15278 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:51.311090 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:51.347396 13811 raft_consensus.cc:1238] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20250623 01:58:51.170570 15149 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:51.254972 15181 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20250623 01:58:51.302788 15275 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:51.335104 13812 rpcz_store.cc:269] 0623 01:58:49.379129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.864909 (+485780us) service_pool.cc:225] Handling call
0623 01:58:51.289379 (+1424470us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54003665d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:51.289499 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":425966336}
I20250623 01:58:51.348690 15235 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:51.229152 15280 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.306385 14922 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:51.289224 15209 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:51.288964 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.255393 15291 consensus_queue.cc:1046] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
I20250623 01:58:51.311784 15281 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250623 01:58:51.361786 15247 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.323s user 0.003s sys 0.013s
I20250623 01:58:51.351817 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:51.334132 15073 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:51.342296 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:51.340324 15246 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:51.348476 14886 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:51.379176 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.384296 14478 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.277909 15302 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250623 01:58:51.378700 15301 log.cc:927] Time spent T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.214s user 0.000s sys 0.000s
I20250623 01:58:51.363947 15235 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:51.391227 15293 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.246s user 0.001s sys 0.000s
I20250623 01:58:51.350225 15163 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:51.393431 15247 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.380s user 0.004s sys 0.014s
W20250623 01:58:51.419572 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 116848us
I20250623 01:58:51.379935 15085 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:51.354087 15219 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.400810 15291 consensus_queue.cc:1230] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer 85a58a834b6b4b0dbe197b3dad87e382 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250623 01:58:51.422570 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.422760 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.420236 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1182) took 2169 ms. Trace:
I20250623 01:58:51.400023 15073 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:51.473356 13886 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:51.424664 13961 rpcz_store.cc:276] 0623 01:58:49.251330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.234172 (+1982842us) service_pool.cc:225] Handling call
0623 01:58:51.234442 (+ 270us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:51.276409 (+ 41967us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.276440 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:51.278465 (+ 2025us) log.cc:844] Serialized 26 byte log entry
0623 01:58:51.417125 (+138660us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:51.417173 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.417261 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:51.420043 (+ 2782us) raft_consensus.cc:1683] finished
0623 01:58:51.420059 (+ 16us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.420181 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41638,"raft.queue_time_us":76018,"spinlock_wait_cycles":18015616,"thread_start_us":63220,"threads_started":1,"wal-append.queue_time_us":91201}
I20250623 01:58:51.381407 15209 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.477386 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:51.486014 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 57726.2us
I20250623 01:58:51.415019 14948 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:58:51.495129 13852 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:51.502585 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.392557 15237 consensus_queue.cc:1046] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20250623 01:58:51.509370 13925 leader_election.cc:400] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:51.422508 15178 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:51.512446 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.517768 13880 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:51.518373 13880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 559) took 1257 ms. Trace:
I20250623 01:58:51.518724 13880 rpcz_store.cc:276] 0623 01:58:50.260900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.474980 (+1214080us) service_pool.cc:225] Handling call
0623 01:58:51.518303 (+ 43323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:51.364773 15275 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.565026 13925 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:51.402755 15251 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:51.569208 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.422540 13881 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:51.571035 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.508419 15306 raft_consensus.cc:2802] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:51.473212 15085 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:51.575307 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.574563 13925 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:51.580772 13925 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.576107 15306 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:51.578383 15163 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:51.512046 15112 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:51.581691 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.583511 13959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1162) took 2920 ms. Trace:
I20250623 01:58:51.574280 15246 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:51.583089 15040 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:51.579499 15316 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:51.588912 15237 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:51.589603 13880 raft_consensus.cc:2391] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:51.589706 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.651458 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.651718 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:51.652277 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.652422 13811 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:51.653008 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.653723 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.654342 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.654934 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.590346 13959 rpcz_store.cc:276] 0623 01:58:48.663207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.229304 (+1566097us) service_pool.cc:225] Handling call
0623 01:58:50.229669 (+ 365us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:50.230083 (+ 414us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:50.230125 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:50.247768 (+ 17643us) log.cc:844] Serialized 26 byte log entry
0623 01:58:51.157784 (+910016us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:51.157827 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.157889 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:51.583298 (+425409us) raft_consensus.cc:1683] finished
0623 01:58:51.583320 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.583457 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6230,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":12548,"thread_start_us":856081,"threads_started":1,"wal-append.queue_time_us":816143}
I20250623 01:58:51.577553 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.655627 13961 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:51.655810 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.656411 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:51.656876 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.657436 13959 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:51.658154 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.658321 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.424695 15277 consensus_queue.cc:237] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:51.583581 15311 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20250623 01:58:51.576495 14886 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:51.599630 15307 log.cc:927] Time spent T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.176s user 0.000s sys 0.000s
I20250623 01:58:51.669672 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:51.601835 13957 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 1082) took 2214 ms (client timeout 1858 ms). Trace:
I20250623 01:58:51.603646 14926 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.606163 13960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1175) took 2726 ms. Trace:
I20250623 01:58:51.675174 13960 rpcz_store.cc:276] 0623 01:58:48.879895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.187702 (+2307807us) service_pool.cc:225] Handling call
0623 01:58:51.187971 (+ 269us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.188170 (+ 199us) raft_consensus.cc:1532] Early marking committed up to index 2
0623 01:58:51.248676 (+ 60506us) log.cc:844] Serialized 14 byte log entry
0623 01:58:51.604187 (+355511us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b1800340fc0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b4061f 00007f41e0b428c1 00007f41e0b42879 00007f41e0b42809 00007f41e0b427d1 00007f41e0b41acd 00007f41e9b9b124 00007f41e9b9af59 00007f41e0b23646
0623 01:58:51.605693 (+ 1506us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b1800340fc0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e36eb2ec 00007f41e0b2325c 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3d565 00007f41e3a96c50 00007f41e3aa92f6 00007f41e3ab9e67
0623 01:58:51.605852 (+ 159us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.605899 (+ 47us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:51.605936 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.605997 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.606103 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61729,"prepare.queue_time_us":174780,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":44617,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":9964,"spinlock_wait_cycles":23281664,"thread_start_us":97596,"threads_started":3,"wal-append.queue_time_us":89471}
I20250623 01:58:51.566278 15178 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.583192 15038 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:51.607190 13954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1180) took 2356 ms. Trace:
I20250623 01:58:51.629665 15040 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:51.679394 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:51.596966 15163 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:51.634377 13879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 542) took 1575 ms (client timeout 1786 ms). Trace:
I20250623 01:58:51.587415 15306 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:51.640341 15244 consensus_meta.cc:287] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.538s user 0.006s sys 0.006s
I20250623 01:58:51.640568 15132 raft_consensus.cc:2747] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:58:51.640718 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 781906us
I20250623 01:58:51.687170 13807 raft_consensus.cc:2408] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:51.641090 15219 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:51.581027 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 106820us
I20250623 01:58:51.641394 15012 raft_consensus.cc:3053] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:51.642364 15085 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:51.642544 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1169) took 1339 ms. Trace:
I20250623 01:58:51.643532 13806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1170) took 1339 ms. Trace:
I20250623 01:58:51.700613 13806 rpcz_store.cc:276] 0623 01:58:50.303820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.304298 (+ 478us) service_pool.cc:225] Handling call
0623 01:58:50.659625 (+355327us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.201636 (+542011us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.643306 (+441670us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54001450d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:51.643476 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1183294208}
I20250623 01:58:51.645182 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:51.645872 15280 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.614455 15246 raft_consensus.cc:695] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:58:51.646406 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 394731us
I20250623 01:58:51.705878 13779 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:58:51.648412 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.582655 13925 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:51.658819 13886 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:51.699374 13809 rpcz_store.cc:276] 0623 01:58:50.303566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.303693 (+ 127us) service_pool.cc:225] Handling call
0623 01:58:50.303971 (+ 278us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:50.304142 (+ 171us) raft_consensus.cc:1532] Early marking committed up to index 3
0623 01:58:50.455527 (+151385us) log.cc:844] Serialized 14 byte log entry
0623 01:58:51.642033 (+1186506us) log.cc:844] Serialized 14 byte log entry
0623 01:58:51.642208 (+ 175us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.642254 (+ 46us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:51.642303 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.642372 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.642489 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":628550,"prepare.queue_time_us":164479,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":556583,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":107,"thread_start_us":629923,"threads_started":1,"wal-append.queue_time_us":627003}
I20250623 01:58:51.707365 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:51.708542 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 125953us
I20250623 01:58:51.681541 15291 consensus_queue.cc:1046] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20250623 01:58:51.711323 13925 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:51.711984 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.712563 13809 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:51.665357 13959 raft_consensus.cc:2391] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:51.627658 15316 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.674515 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.682578 14922 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:51.677809 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.630007 15273 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.680001 13960 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:51.679715 13954 rpcz_store.cc:276] 0623 01:58:49.250624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.233374 (+1982750us) service_pool.cc:225] Handling call
0623 01:58:51.233664 (+ 290us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:51.290000 (+ 56336us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.290032 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:51.310655 (+ 20623us) log.cc:844] Serialized 26 byte log entry
0623 01:58:51.420971 (+110316us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:51.420998 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.421044 (+ 46us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:51.606947 (+185903us) raft_consensus.cc:1683] finished
0623 01:58:51.606966 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.607123 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50285,"raft.queue_time_us":71241,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":23185,"thread_start_us":56432,"threads_started":1,"wal-append.queue_time_us":215179}
I20250623 01:58:51.717252 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.717909 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:58:51.673286 13957 rpcz_store.cc:269] 0623 01:58:49.387321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.908365 (+521044us) service_pool.cc:225] Handling call
0623 01:58:51.601639 (+1693274us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b5400074254. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:51.601772 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":595718912}
W20250623 01:58:51.689126 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.669354 14886 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.673632 15190 consensus_queue.cc:1046] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20250623 01:58:51.703114 15209 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:51.667537 15237 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:51.709775 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:51.684583 15038 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:51.729009 13779 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.725968 15284 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:51.713737 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.717161 15163 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:51.715068 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.682186 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:51.721284 13776 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.723613 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:51.681795 13879 rpcz_store.cc:269] 0623 01:58:50.058659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.069777 (+ 11118us) service_pool.cc:225] Handling call
0623 01:58:51.634199 (+1564422us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b5400389ad4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:51.634319 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2039825664}
I20250623 01:58:51.726362 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:51.738343 15065 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:51.717200 15132 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:51.737733 13960 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
W20250623 01:58:51.814477 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 93258.4us
I20250623 01:58:51.814797 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.685998 15040 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.739439 13959 raft_consensus.cc:2391] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:51.713779 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.817212 13809 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:51.803018 15302 raft_consensus.cc:3053] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:51.818250 13809 raft_consensus.cc:1238] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:51.664566 13811 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:51.819656 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.737010 13779 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:58:51.821766 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 84807.6us
I20250623 01:58:51.737450 15228 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:51.805198 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.738507 15275 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:51.733474 15244 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250623 01:58:51.832588 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 158072us
I20250623 01:58:51.746251 15163 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:51.808877 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.808903 13954 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:51.699087 15012 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:51.810940 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.887652 15012 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.815598 13776 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.888535 13880 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:51.888667 15012 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:51.888990 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 73482.4us
I20250623 01:58:51.889423 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.889756 13776 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:51.805240 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.898075 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.898761 13880 raft_consensus.cc:2366] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:51.899626 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.822890 13779 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:58:51.797595 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.799247 15284 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:51.844295 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:51.841971 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.848585 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:51.848884 15163 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:51.849221 14922 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:51.849980 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.851559 15073 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.853514 15038 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:51.907837 15126 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:51.829752 15065 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:58:51.825260 15228 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:51.907984 15240 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:51.857208 15286 consensus_queue.cc:1046] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:58:51.814309 15132 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:51.990689 15132 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:51.858736 14886 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.859925 15316 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.833360 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1168) took 1527 ms. Trace:
I20250623 01:58:51.862286 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 1185) took 1138 ms. Trace:
I20250623 01:58:51.900985 15291 consensus_queue.cc:1046] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.088s
I20250623 01:58:51.877326 15237 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:51.878108 15326 log.cc:927] Time spent T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20250623 01:58:51.819238 15302 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:58:51.880363 15268 log.cc:927] Time spent T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 1.579s user 0.001s sys 0.015s
I20250623 01:58:51.886559 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:51.812057 15270 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:51.825456 13886 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:51.856756 15235 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:51.906854 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 84047.6us
I20250623 01:58:52.001271 13779 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:52.001755 15302 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.907528 13776 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.002522 13779 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:51.910156 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.907999 13954 raft_consensus.cc:2408] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:52.004464 14512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48412 (request call id 69) took 1319 ms. Trace:
I20250623 01:58:52.004964 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.004833 14512 rpcz_store.cc:276] 0623 01:58:50.685559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.689948 (+ 4389us) service_pool.cc:225] Handling call
0623 01:58:50.690312 (+ 364us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.035941 (+345629us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.004235 (+968294us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b5400490cd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:52.004406 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72578944}
I20250623 01:58:52.005769 13954 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:52.006301 15131 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:52.006601 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.006894 15284 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:52.007097 15290 raft_consensus.cc:3053] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:52.007185 13954 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:52.007477 15290 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:52.008564 15073 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.008337 15290 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.011564 13812 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
W20250623 01:58:52.012827 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 105406us
I20250623 01:58:52.013571 13776 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.013736 13779 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:52.003540 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.014838 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1102) took 1816 ms. Trace:
I20250623 01:58:52.015015 13954 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:52.015164 13882 rpcz_store.cc:276] 0623 01:58:50.198726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.231977 (+ 33251us) service_pool.cc:225] Handling call
0623 01:58:52.003396 (+1771419us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:52.014782 (+ 11386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6877440}
I20250623 01:58:52.015920 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.016103 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.016575 13954 raft_consensus.cc:2391] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:52.016736 13882 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:51.911082 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:51.979166 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.980882 13962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1177) took 2825 ms. Trace:
W20250623 01:58:51.981593 15095 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.968s user 0.023s sys 0.028s
I20250623 01:58:51.809495 15246 consensus_queue.cc:237] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:51.983093 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 45937.1us
I20250623 01:58:52.019244 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:51.982982 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:51.883885 15178 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:51.900605 15276 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:51.901268 14926 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:51.841972 15244 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:51.900568 13957 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:51.908263 15243 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:51.988813 15240 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:51.990204 15228 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.018965 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:51.994510 13807 rpcz_store.cc:276] 0623 01:58:50.723835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.085957 (+362122us) service_pool.cc:225] Handling call
0623 01:58:51.862059 (+776102us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b5400366fd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:51.862228 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1226534272}
I20250623 01:58:51.875566 15273 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:52.064488 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:51.986449 15065 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:51.991608 15045 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:52.017308 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.017629 13811 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:51.940281 15306 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:52.021178 13779 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:52.019654 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1131) took 1182 ms. Trace:
I20250623 01:58:52.022152 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.071540 13882 rpcz_store.cc:276] 0623 01:58:50.837690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.015891 (+1178201us) service_pool.cc:225] Handling call
0623 01:58:52.019599 (+ 3708us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.071764 13959 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:52.072554 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.072710 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.025426 13961 raft_consensus.cc:2391] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:52.073112 13882 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:52.073810 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.074353 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.074898 13957 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:52.075306 13961 raft_consensus.cc:1238] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:58:52.075871 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1074) took 2691 ms. Trace:
I20250623 01:58:52.076190 13961 rpcz_store.cc:276] 0623 01:58:49.384589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.074769 (+2690180us) service_pool.cc:225] Handling call
0623 01:58:52.075080 (+ 311us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:52.075647 (+ 567us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.075825 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.067301 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:52.071017 14882 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.078203 15045 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:52.078684 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.079461 13884 raft_consensus.cc:2366] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
W20250623 01:58:52.079890 13884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 536) took 2022 ms (client timeout 1820 ms). Trace:
I20250623 01:58:52.080150 15045 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:52.080307 13884 rpcz_store.cc:269] 0623 01:58:50.056938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.069839 (+ 12901us) service_pool.cc:225] Handling call
0623 01:58:52.078541 (+2008702us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:52.079772 (+ 1231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61535104}
I20250623 01:58:52.017804 13962 rpcz_store.cc:276] 0623 01:58:49.155699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.218727 (+2063028us) service_pool.cc:225] Handling call
0623 01:58:51.219019 (+ 292us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.219198 (+ 179us) raft_consensus.cc:1532] Early marking committed up to index 3
0623 01:58:51.582492 (+363294us) log.cc:844] Serialized 14 byte log entry
0623 01:58:51.980406 (+397914us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.980493 (+ 87us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:51.980553 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.980637 (+ 84us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.980796 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39304,"prepare.queue_time_us":187484,"prepare.run_cpu_time_us":7183,"prepare.run_wall_time_us":60643,"raft.queue_time_us":34221,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":15158,"thread_start_us":678878,"threads_started":3,"wal-append.queue_time_us":128824}
I20250623 01:58:52.011229 15290 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:52.021515 13776 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.028465 15244 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
W20250623 01:58:51.993638 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 159843us
W20250623 01:58:52.062031 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:52.025236 15131 raft_consensus.cc:2747] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.019343 15181 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:52.019063 15284 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:51.985229 15126 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:52.065802 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:52.068236 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.028062 15270 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:52.029137 15302 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:52.073859 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.105178 13882 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:52.105769 13852 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:52.106101 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.106675 13882 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:52.106645 13852 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:52.107246 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1141) took 1249 ms. Trace:
I20250623 01:58:52.107542 13882 rpcz_store.cc:276] 0623 01:58:50.858227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.105870 (+1247643us) service_pool.cc:225] Handling call
0623 01:58:52.107199 (+ 1329us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.108196 13852 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:52.108325 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.108897 13882 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:52.084084 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:52.084431 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 62998.2us
I20250623 01:58:52.110111 13954 raft_consensus.cc:2408] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250623 01:58:52.084615 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 63511.5us
I20250623 01:58:52.111269 13776 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.088346 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1189) took 1268 ms. Trace:
I20250623 01:58:52.111750 13779 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:52.104580 15126 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:52.030102 15235 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.089535 15276 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.113512 13813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1172) took 1809 ms. Trace:
I20250623 01:58:52.088507 15243 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.070188 15065 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:52.090044 14509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48412 (request call id 72) took 1062 ms. Trace:
I20250623 01:58:52.120745 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.092186 13810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1190) took 1258 ms. Trace:
I20250623 01:58:51.994350 13805 rpcz_store.cc:276] 0623 01:58:50.303241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.316163 (+ 12922us) service_pool.cc:225] Handling call
0623 01:58:50.645621 (+329458us) raft_consensus.cc:1469] Updating replica for 2 ops
0623 01:58:50.934035 (+288414us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:50.968728 (+ 34693us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0623 01:58:51.086280 (+117552us) log.cc:844] Serialized 52 byte log entry
0623 01:58:51.298159 (+211879us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:51.363125 (+ 64966us) log.cc:844] Serialized 14 byte log entry
0623 01:58:51.675773 (+312648us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.679732 (+ 3959us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:51.814034 (+134302us) raft_consensus.cc:1683] finished
0623 01:58:51.814078 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:51.830335 (+ 16257us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":482178,"prepare.queue_time_us":43752,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":2219,"raft.queue_time_us":438208,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":38360,"thread_start_us":118661,"threads_started":1,"wal-append.queue_time_us":270236}
I20250623 01:58:52.097193 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:52.070837 15178 consensus_queue.cc:1046] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20250623 01:58:52.098049 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1157) took 3435 ms. Trace:
I20250623 01:58:52.130147 13958 rpcz_store.cc:276] 0623 01:58:48.662159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.148922 (+486763us) service_pool.cc:225] Handling call
0623 01:58:49.149347 (+ 425us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:49.327725 (+178378us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:49.327760 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:49.383920 (+ 56160us) log.cc:844] Serialized 26 byte log entry
0623 01:58:49.982864 (+598944us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:49.982910 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:49.982989 (+ 79us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:51.879842 (+1896853us) spinlock_profiling.cc:243] Waited 56.3 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:58:52.085788 (+205946us) raft_consensus.cc:1683] finished
0623 01:58:52.085808 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.097989 (+ 12181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9594,"raft.queue_time_us":41356,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":6183,"spinlock_wait_cycles":129377024,"thread_start_us":648170,"threads_started":2,"wal-append.queue_time_us":819551}
I20250623 01:58:52.131319 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.100224 15181 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.089982 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:52.099167 13881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1115) took 1801 ms (client timeout 2225 ms). Trace:
I20250623 01:58:52.102928 13880 raft_consensus.cc:2391] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:58:52.022126 15095 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 2.012s user 0.023s sys 0.028s
I20250623 01:58:52.113934 13813 rpcz_store.cc:276] 0623 01:58:50.304318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.682775 (+378457us) service_pool.cc:225] Handling call
0623 01:58:50.726214 (+ 43439us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:50.726445 (+ 231us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:50.726487 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:50.726548 (+ 61us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:50.726592 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:50.726666 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.113356 (+1386690us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b54001446d0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:52.113457 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99027200}
I20250623 01:58:52.223392 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.121609 15270 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:52.224323 13813 raft_consensus.cc:2391] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:52.114581 13885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1112) took 1890 ms. Trace:
I20250623 01:58:52.115401 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.226974 13882 raft_consensus.cc:2391] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:52.115729 14478 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:52.116945 13852 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:52.119477 13776 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.111898 13808 rpcz_store.cc:276] 0623 01:58:50.820096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.238446 (+418350us) service_pool.cc:225] Handling call
0623 01:58:51.238702 (+ 256us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:51.684897 (+446195us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.684931 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:51.688124 (+ 3193us) log.cc:844] Serialized 26 byte log entry
0623 01:58:51.923003 (+234879us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:51.923064 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.923162 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:52.088206 (+165044us) raft_consensus.cc:1683] finished
0623 01:58:52.088219 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.088306 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":449879,"raft.queue_time_us":445992,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8295,"thread_start_us":202053,"threads_started":1,"wal-append.queue_time_us":309219}
W20250623 01:58:52.228941 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 112082us
I20250623 01:58:52.229563 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.230854 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.123646 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.232525 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.233737 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.234985 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.121474 13810 rpcz_store.cc:276] 0623 01:58:50.833676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.239620 (+405944us) service_pool.cc:225] Handling call
0623 01:58:51.275840 (+ 36220us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:51.646966 (+371126us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.647000 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:51.647486 (+ 486us) log.cc:844] Serialized 26 byte log entry
0623 01:58:51.981565 (+334079us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:51.981634 (+ 69us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.981731 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:51.990862 (+ 9131us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b48000315d8. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e8956871 00007f41e894da17 00007f41e894d8aa 00007f41e8a4297f 00007f41e8a42647 00007f41e8a56c6c 00007f41e8a56c18 00007f41e8a56b98 00007f41e8a56b53 00007f41e8a55dfc 00007f41e79dc736
0623 01:58:52.091938 (+101076us) raft_consensus.cc:1683] finished
0623 01:58:52.091957 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.092127 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":370769,"raft.queue_time_us":390528,"raft.run_cpu_time_us":861,"raft.run_wall_time_us":320974,"spinlock_wait_cycles":2597760,"thread_start_us":333658,"threads_started":1,"wal-append.queue_time_us":351890}
I20250623 01:58:52.126140 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.236413 13807 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.128610 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.132583 13809 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250623 01:58:52.238734 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.127090 15271 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:52.182839 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.249950 13806 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:52.124431 15235 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:52.184815 13953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 98) took 3030 ms. Trace:
I20250623 01:58:52.121029 14509 rpcz_store.cc:276] 0623 01:58:51.028030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.028217 (+ 187us) service_pool.cc:225] Handling call
0623 01:58:52.089826 (+1061609us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b540056dbd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:52.089973 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2217448320}
I20250623 01:58:52.116880 15337 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:52.129637 13960 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37424
I20250623 01:58:52.116990 15073 raft_consensus.cc:2747] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.127234 15131 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:52.250380 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 131008us
I20250623 01:58:52.306144 13776 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:52.251135 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 135481us
I20250623 01:58:52.263111 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.228056 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.287793 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.284332 13852 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:52.224745 13885 rpcz_store.cc:276] 0623 01:58:50.224444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.260308 (+ 35864us) service_pool.cc:225] Handling call
0623 01:58:50.260562 (+ 254us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:50.662769 (+402207us) raft_consensus.cc:1532] Early marking committed up to index 2
0623 01:58:50.878235 (+215466us) log.cc:844] Serialized 14 byte log entry
0623 01:58:50.904194 (+ 25959us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:50.904263 (+ 69us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:50.904318 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.114366 (+1210048us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b540018e054. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:52.114412 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.114531 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":414838,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":2352,"raft.queue_time_us":14769,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":124,"spinlock_wait_cycles":924230912,"thread_start_us":196922,"threads_started":2,"wal-append.queue_time_us":257054}
I20250623 01:58:52.310487 13885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1120) took 1957 ms. Trace:
I20250623 01:58:52.310822 13885 rpcz_store.cc:276] 0623 01:58:50.352747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.309804 (+1957057us) service_pool.cc:225] Handling call
0623 01:58:52.310080 (+ 276us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.310229 (+ 149us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.310251 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.310287 (+ 36us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:52.310312 (+ 25us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.310357 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.310441 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.311457 13807 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:52.284912 13811 raft_consensus.cc:2391] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:52.289718 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:52.291042 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:52.126979 15284 consensus_queue.cc:1046] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.098s
I20250623 01:58:52.288911 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.283680 14970 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:52.222714 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.329802 13956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1174) took 3497 ms. Trace:
I20250623 01:58:52.292052 13960 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:58:52.287803 15235 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:52.287323 14512 raft_consensus.cc:2391] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 5: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:52.312381 13885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1123) took 1927 ms. Trace:
I20250623 01:58:52.349874 13885 rpcz_store.cc:276] 0623 01:58:50.384973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.311700 (+1926727us) service_pool.cc:225] Handling call
0623 01:58:52.311982 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.312126 (+ 144us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.312148 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.312185 (+ 37us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:52.312211 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.312255 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.312337 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.351928 13885 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250623 01:58:52.352653 13885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1139) took 1495 ms. Trace:
I20250623 01:58:52.307024 15250 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:52.328166 15280 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:52.287174 15276 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:58:52.319985 13925 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.363003 13806 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:52.291803 15316 consensus_queue.cc:1046] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
W20250623 01:58:52.218626 13881 rpcz_store.cc:269] 0623 01:58:50.298014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.239661 (+941647us) service_pool.cc:225] Handling call
0623 01:58:52.098980 (+859319us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b54003d72d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:52.099108 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75399424}
I20250623 01:58:52.303836 15240 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.314390 13813 raft_consensus.cc:2374] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3. Re-sending same reply.
I20250623 01:58:52.280441 15244 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.314630 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1140) took 1456 ms. Trace:
I20250623 01:58:52.287016 15325 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:52.353039 13885 rpcz_store.cc:276] 0623 01:58:50.857594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.351199 (+1493605us) service_pool.cc:225] Handling call
0623 01:58:52.351601 (+ 402us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:52.352360 (+ 759us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.352591 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.372931 13885 raft_consensus.cc:1238] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:52.358831 13960 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.302286 15337 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:52.373687 13885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 82) took 1225 ms. Trace:
I20250623 01:58:52.330369 13956 rpcz_store.cc:276] 0623 01:58:48.832431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.181888 (+2349457us) service_pool.cc:225] Handling call
0623 01:58:51.182277 (+ 389us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.182530 (+ 253us) raft_consensus.cc:1532] Early marking committed up to index 2
0623 01:58:51.305351 (+122821us) log.cc:844] Serialized 14 byte log entry
0623 01:58:52.328718 (+1023367us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b18001af440. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e36eb2ec 00007f41e0b2325c 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e36dd04b 00007f41e36e2b53 00007f41e36e306a
0623 01:58:52.329267 (+ 549us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.329372 (+ 105us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:52.329432 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.329596 (+ 164us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.329734 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":297,"prepare.queue_time_us":128704,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":6129,"raft.queue_time_us":175,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":14578,"spinlock_wait_cycles":4432128,"thread_start_us":163511,"threads_started":2,"wal-append.queue_time_us":116903}
I20250623 01:58:52.332558 15272 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:52.348901 15235 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:52.374992 13956 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:52.375854 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 81) took 1236 ms. Trace:
I20250623 01:58:52.376250 13881 rpcz_store.cc:276] 0623 01:58:51.139484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.365538 (+1226054us) service_pool.cc:225] Handling call
0623 01:58:52.365875 (+ 337us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.366041 (+ 166us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.366082 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.367907 (+ 1825us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:58:52.371880 (+ 3973us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.375687 (+ 3807us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.375783 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.377915 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 83) took 1229 ms. Trace:
I20250623 01:58:52.344514 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.249624 15271 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:52.348506 14970 raft_consensus.cc:695] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:52.369539 13882 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:58:52.367990 15177 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:52.355691 15073 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:52.373883 15337 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:52.337865 15306 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:52.363714 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 43848us
I20250623 01:58:52.423411 13925 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.425683 13925 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:52.428304 13925 leader_election.cc:400] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:52.428668 13925 leader_election.cc:403] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:52.372000 15325 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:52.417719 13957 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:52.371228 14948 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
W20250623 01:58:52.464452 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 158383us
I20250623 01:58:52.466114 13776 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.412428 15271 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:52.380658 13956 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250623 01:58:52.364003 15131 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.368351 15250 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:52.399322 15012 consensus_meta.cc:287] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.510s user 0.010s sys 0.000s
I20250623 01:58:52.411392 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1161) took 3748 ms. Trace:
I20250623 01:58:52.445770 15325 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.411577 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:52.415426 13960 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.287297 13953 rpcz_store.cc:276] 0623 01:58:49.154866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.208045 (+2053179us) service_pool.cc:225] Handling call
0623 01:58:51.215474 (+ 7429us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.245246 (+ 29772us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.249399 (+ 4153us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.255527 (+ 6128us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:58:51.278022 (+ 22495us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.297668 (+ 19646us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.133481 (+835813us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b54003f7fd0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:52.184751 (+ 51270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8655232}
I20250623 01:58:52.500744 13960 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:52.502512 15346 log.cc:927] Time spent T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250623 01:58:52.503876 15279 raft_consensus.cc:991] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:58:52.504360 15012 raft_consensus.cc:1079] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:58:52.505007 14508 raft_consensus.cc:1238] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:52.506279 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.506662 15334 consensus_queue.cc:1046] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:58:52.379248 15280 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:52.507555 15280 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:52.508401 15280 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.472443 14970 consensus_queue.cc:237] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.372227 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:52.475822 15316 raft_consensus.cc:3053] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:52.476321 15302 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:52.489774 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:52.477813 15065 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.471808 13776 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.478838 14512 raft_consensus.cc:1238] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:52.480715 15243 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.484247 15240 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.442533 15338 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:58:52.422224 15344 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.095s user 0.000s sys 0.002s
I20250623 01:58:52.488698 13956 raft_consensus.cc:1238] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:52.370607 13884 rpcz_store.cc:276] 0623 01:58:50.857898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.081188 (+1223290us) service_pool.cc:225] Handling call
0623 01:58:52.305303 (+224115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.378446 13885 rpcz_store.cc:276] 0623 01:58:51.148463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.372342 (+1223879us) service_pool.cc:225] Handling call
0623 01:58:52.372702 (+ 360us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.373413 (+ 711us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.373629 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.448058 15040 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:52.493808 13955 rpcz_store.cc:276] 0623 01:58:48.663008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.223258 (+1560250us) service_pool.cc:225] Handling call
0623 01:58:51.602258 (+1379000us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:51.602431 (+ 173us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:51.602455 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:51.602492 (+ 37us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:51.602523 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:51.602570 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.393724 (+791154us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b5400074250. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:52.411338 (+ 17614us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3170260224}
I20250623 01:58:52.370612 15270 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.507005 13883 raft_consensus.cc:2366] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
W20250623 01:58:52.550562 13883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1126) took 1920 ms (client timeout 2146 ms). Trace:
W20250623 01:58:52.551054 13883 rpcz_store.cc:269] 0623 01:58:50.630205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.266942 (+636737us) service_pool.cc:225] Handling call
0623 01:58:52.506139 (+1239197us) spinlock_profiling.cc:243] Waited 9.03 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:52.550485 (+ 44346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20764544}
I20250623 01:58:52.415589 13805 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:52.502537 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.511790 13960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:37424 (request call id 1087) took 1250 ms. Trace:
I20250623 01:58:52.553112 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.553334 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.554318 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.554849 13880 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:58:52.555536 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.486131 15181 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:52.442284 15178 raft_consensus.cc:2762] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20250623 01:58:52.516197 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 44470.5us
I20250623 01:58:52.442811 15034 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:52.421633 15177 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.421810 15163 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:52.521835 15345 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20250623 01:58:52.429963 15073 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:52.532829 15045 consensus_queue.cc:1046] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20250623 01:58:52.455641 15132 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.570937 15040 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.574533 13812 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:52.575937 15040 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:52.381961 13881 rpcz_store.cc:276] 0623 01:58:51.148742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.377187 (+1228445us) service_pool.cc:225] Handling call
0623 01:58:52.377462 (+ 275us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.377616 (+ 154us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.377640 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.377680 (+ 40us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:58:52.377712 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.377775 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.377868 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.513921 15316 raft_consensus.cc:738] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:52.544205 13956 raft_consensus.cc:1238] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:52.546979 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:52.516417 15280 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.517201 15131 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:58:52.558393 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:52.562577 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.496907 15228 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.568960 13960 rpcz_store.cc:276] 0623 01:58:51.275244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.103984 (+828740us) service_pool.cc:225] Handling call
0623 01:58:52.501313 (+397329us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.600805 13959 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:52.570396 14507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48398 (request call id 103) took 1347 ms. Trace:
I20250623 01:58:52.573230 15065 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.581084 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1075) took 3196 ms. Trace:
I20250623 01:58:52.606541 13961 rpcz_store.cc:276] 0623 01:58:49.384894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.076816 (+2691922us) service_pool.cc:225] Handling call
0623 01:58:52.077112 (+ 296us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:52.085717 (+ 8605us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.085746 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:52.091493 (+ 5747us) log.cc:844] Serialized 26 byte log entry
0623 01:58:52.284048 (+192555us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:52.284115 (+ 67us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.284244 (+ 129us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:52.580812 (+296568us) raft_consensus.cc:1683] finished
0623 01:58:52.580837 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.581002 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2759,"raft.queue_time_us":7731,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":1466,"thread_start_us":192059,"threads_started":1,"wal-append.queue_time_us":267628}
I20250623 01:58:52.588302 13885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 579) took 1012 ms. Trace:
I20250623 01:58:52.517830 15302 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.572368 15244 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.578579 13883 raft_consensus.cc:1238] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:52.611696 13960 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:52.614195 13883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 86) took 1448 ms. Trace:
I20250623 01:58:52.614688 13883 rpcz_store.cc:276] 0623 01:58:51.165433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.577950 (+1412517us) service_pool.cc:225] Handling call
0623 01:58:52.578348 (+ 398us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.613928 (+ 35580us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.614126 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.615526 13925 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.617179 15040 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:52.617484 13883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 586) took 1040 ms. Trace:
I20250623 01:58:52.617928 13883 rpcz_store.cc:276] 0623 01:58:51.577522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.616730 (+1039208us) service_pool.cc:225] Handling call
0623 01:58:52.616994 (+ 264us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.617165 (+ 171us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.617191 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.617230 (+ 39us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:52.617264 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.617339 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.617432 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.619418 13954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1097) took 1342 ms. Trace:
I20250623 01:58:52.611240 13885 rpcz_store.cc:276] 0623 01:58:51.575897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.586908 (+1011011us) service_pool.cc:225] Handling call
0623 01:58:52.587270 (+ 362us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.587486 (+ 216us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.587524 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.587579 (+ 55us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:58:52.588007 (+ 428us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.588103 (+ 96us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.588236 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:58:52.612082 13953 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:52.615273 15316 consensus_queue.cc:260] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:52.619956 13954 rpcz_store.cc:276] 0623 01:58:51.277268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.112048 (+834780us) service_pool.cc:225] Handling call
0623 01:58:52.112416 (+ 368us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:52.112665 (+ 249us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.112702 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:52.113263 (+ 561us) log.cc:844] Serialized 26 byte log entry
0623 01:58:52.345618 (+232355us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:52.345654 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.345722 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:52.619221 (+273499us) raft_consensus.cc:1683] finished
0623 01:58:52.619243 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.619368 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":159205,"threads_started":1,"wal-append.queue_time_us":481787}
I20250623 01:58:52.619410 13883 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250623 01:58:52.616267 15290 consensus_meta.cc:287] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.533s user 0.005s sys 0.006s
I20250623 01:58:52.626188 13883 raft_consensus.cc:1238] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:52.627925 13925 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:52.605508 14507 rpcz_store.cc:276] 0623 01:58:51.222717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.229072 (+ 6355us) service_pool.cc:225] Handling call
0623 01:58:52.468117 (+1239045us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b58001e2a98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:52.562137 (+ 94020us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4206848}
I20250623 01:58:52.619498 15126 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.632958 15316 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:52.637362 15178 raft_consensus.cc:2762] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250623 01:58:52.654062 15228 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:52.654243 15302 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.654857 15177 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.656745 15337 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:52.658136 15313 log.cc:927] Time spent T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.216s user 0.002s sys 0.000s
I20250623 01:58:52.658339 15337 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.660431 15250 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.661689 15163 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:52.661603 15251 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:52.666613 15272 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:52.667243 15272 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.667491 15271 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.669636 15271 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.670603 15252 raft_consensus.cc:3053] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:52.671113 15252 raft_consensus.cc:738] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:52.671340 15235 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.672644 15235 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.672168 15252 consensus_queue.cc:260] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:52.668596 15325 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:52.679633 15336 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.005s sys 0.004s
W20250623 01:58:52.680645 15336 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s user 0.005s sys 0.004s
I20250623 01:58:52.680460 15351 consensus_queue.cc:1046] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:58:52.682725 15350 consensus_queue.cc:1046] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:52.570478 13776 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:52.709592 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 92944.7us
I20250623 01:58:52.801622 14882 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:58:52.802450 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 231966us
I20250623 01:58:52.678872 15252 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:58:52.712574 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:52.712836 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:52.718520 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 339083us
I20250623 01:58:52.720191 15126 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.651531 15132 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.726197 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1098) took 1448 ms. Trace:
I20250623 01:58:52.726928 13888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1103) took 2527 ms. Trace:
I20250623 01:58:52.733601 15325 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:52.683162 15358 raft_consensus.cc:3053] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:52.813666 15358 raft_consensus.cc:738] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:52.814047 13957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1193) took 1137 ms. Trace:
I20250623 01:58:52.815212 15358 consensus_queue.cc:260] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:52.814734 13957 rpcz_store.cc:276] 0623 01:58:51.676384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.483686 (+807302us) service_pool.cc:225] Handling call
0623 01:58:52.484028 (+ 342us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.569036 (+ 85008us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.582365 (+ 13329us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.583891 (+ 1526us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:52.594420 (+ 10529us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.813341 (+218921us) spinlock_profiling.cc:243] Waited 56.7 ms on lock 0x7b54000bbdd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:52.813437 (+ 96us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.813783 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":130477952}
I20250623 01:58:52.818783 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:52.736058 13880 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:52.736477 15073 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:52.739425 14508 raft_consensus.cc:1238] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4. Ops: []
W20250623 01:58:52.741456 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 292956us
I20250623 01:58:52.806700 13888 rpcz_store.cc:276] 0623 01:58:50.199082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:50.276642 (+ 77560us) service_pool.cc:225] Handling call
0623 01:58:50.347627 (+ 70985us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:51.319566 (+971939us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.726146 (+1406580us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b540029a9d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:52.726650 (+ 504us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2187305216}
I20250623 01:58:52.743189 13885 raft_consensus.cc:1238] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250623 01:58:52.747057 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.806648 13958 rpcz_store.cc:276] 0623 01:58:51.277520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.131826 (+854306us) service_pool.cc:225] Handling call
0623 01:58:52.132104 (+ 278us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:52.132307 (+ 203us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.132331 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:52.186320 (+ 53989us) log.cc:844] Serialized 26 byte log entry
0623 01:58:52.640613 (+454293us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:52.640668 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.640758 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:52.717264 (+ 76506us) raft_consensus.cc:1683] finished
0623 01:58:52.725391 (+ 8127us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.725861 (+ 470us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":337694,"threads_started":1,"wal-append.queue_time_us":404196}
I20250623 01:58:52.826323 13810 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:52.826701 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.827551 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.827684 13958 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:52.828187 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:52.748399 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.828887 13804 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 3
I20250623 01:58:52.829130 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.756610 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.830052 13811 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:52.831234 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.832481 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.756844 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.833901 13807 raft_consensus.cc:2366] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:52.834897 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.835669 13807 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.836277 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 1133) took 1997 ms. Trace:
I20250623 01:58:52.757915 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:52.838200 14481 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:52.838049 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.838380 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:52.838954 13887 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1138) took 1981 ms (client timeout 1927 ms). Trace:
W20250623 01:58:52.839377 13887 rpcz_store.cc:269] 0623 01:58:50.857267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:51.244984 (+387717us) service_pool.cc:225] Handling call
0623 01:58:52.837705 (+1592721us) spinlock_profiling.cc:243] Waited 7.02 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:52.838879 (+ 1174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16137984}
I20250623 01:58:52.839788 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.836710 13882 rpcz_store.cc:276] 0623 01:58:50.855552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.325143 (+1469591us) service_pool.cc:225] Handling call
0623 01:58:52.325581 (+ 438us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:52.413563 (+ 87982us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.835689 (+422126us) spinlock_profiling.cc:243] Waited 43.7 ms on lock 0x7b540018ea54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:52.836091 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100553600}
I20250623 01:58:52.759204 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.841401 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:52.841630 15198 raft_consensus.cc:3053] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:52.841960 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.843376 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.824705 13708 catalog_manager.cc:5582] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:52.760499 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.845202 13886 raft_consensus.cc:2366] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:52.845661 13886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 577) took 1270 ms. Trace:
I20250623 01:58:52.845088 13708 catalog_manager.cc:5582] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 2, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:52.771286 13955 raft_consensus.cc:1238] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:52.847656 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.776480 15230 consensus_queue.cc:1046] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:52.767163 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.848898 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.842058 15198 raft_consensus.cc:738] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:52.849972 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.844350 15272 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.850841 15198 consensus_queue.cc:260] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:52.780277 13959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 1106) took 1360 ms. Trace:
I20250623 01:58:52.782147 13879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1144) took 1051 ms. Trace:
I20250623 01:58:52.852414 13959 rpcz_store.cc:276] 0623 01:58:51.419993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.072494 (+652501us) service_pool.cc:225] Handling call
0623 01:58:52.779666 (+707172us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b54000a4454. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:52.779990 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1213224320}
I20250623 01:58:52.844605 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:52.855018 13888 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.857637 15034 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:52.858460 15034 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:52.790210 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:52.848737 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:52.808763 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:52.828363 13810 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.748034 13812 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:52.755537 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.803694 13776 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:52.842159 13882 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.845973 13886 rpcz_store.cc:276] 0623 01:58:51.575098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.000537 (+425439us) service_pool.cc:225] Handling call
0623 01:58:52.760119 (+759582us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:52.845607 (+ 85488us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5255552}
I20250623 01:58:52.761837 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.764030 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.853294 13879 rpcz_store.cc:276] 0623 01:58:51.730830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.125909 (+395079us) service_pool.cc:225] Handling call
0623 01:58:52.781631 (+655722us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54001a3954. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:52.781896 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":593162112}
I20250623 01:58:52.850592 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.840539 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:52.861809 13808 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:52.862145 13959 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:52.789543 15271 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:52.935708 14478 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:52.877677 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.922539 13809 raft_consensus.cc:2391] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 5: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:52.902707 13886 raft_consensus.cc:1238] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:52.946147 13885 raft_consensus.cc:2408] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:52.847980 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 87) took 1679 ms. Trace:
I20250623 01:58:52.961861 13808 raft_consensus.cc:2391] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
W20250623 01:58:52.882243 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:52.882048 13806 raft_consensus.cc:2391] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:52.927320 13811 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.848876 15040 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:52.967578 15132 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:52.967510 13925 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:52.921298 15331 raft_consensus.cc:3053] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:52.968184 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:52.968413 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.968606 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.968955 13959 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250623 01:58:52.962561 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:52.962159 15358 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.959993 15198 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:52.883440 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:52.966365 15272 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:52.985154 13882 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:52.882652 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:52.985013 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:52.987532 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:52.993125 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 88473.2us
I20250623 01:58:52.976639 15331 raft_consensus.cc:738] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:52.984320 15106 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250623 01:58:52.945693 15271 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.015552 13879 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:53.016052 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:53.017814 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 214400us
I20250623 01:58:53.020571 13776 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:53.023218 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52444 (request call id 189) took 8209 ms. Trace:
I20250623 01:58:53.025568 15149 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:53.026252 15149 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.028883 15149 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.030553 14478 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.030067 13955 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:53.016225 13887 raft_consensus.cc:1273] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:53.025179 13888 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:53.033685 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.037171 13885 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.042413 13807 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:53.049775 13812 raft_consensus.cc:2408] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:53.047106 13956 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:53.013044 15034 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:53.050601 13810 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:53.053076 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:53.069564 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 583) took 1431 ms. Trace:
I20250623 01:58:53.070143 13881 rpcz_store.cc:276] 0623 01:58:51.576862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.604672 (+1027810us) service_pool.cc:225] Handling call
0623 01:58:52.620641 (+ 15969us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:52.628387 (+ 7746us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.630685 (+ 2298us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:52.723786 (+ 93101us) log.cc:844] Serialized 26 byte log entry
0623 01:58:52.807111 (+ 83325us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:58:52.807969 (+ 858us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.859455 (+ 51486us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:52.984799 (+125344us) raft_consensus.cc:1683] finished
0623 01:58:52.986797 (+ 1998us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:53.006289 (+ 19492us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":144,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7611,"thread_start_us":62246,"threads_started":1,"wal-append.queue_time_us":165968}
I20250623 01:58:52.965085 13884 rpcz_store.cc:276] 0623 01:58:51.168761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.571275 (+1402514us) service_pool.cc:225] Handling call
0623 01:58:52.571540 (+ 265us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:58:52.611122 (+ 39582us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.612986 (+ 1864us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:58:52.615041 (+ 2055us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:58:52.616364 (+ 1323us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.847465 (+231101us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b540011e754. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:52.847543 (+ 78us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:52.847808 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74887424}
I20250623 01:58:53.072841 15277 raft_consensus.cc:2762] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250623 01:58:53.073154 15132 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:53.073920 15132 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:58:53.074330 15359 log.cc:927] Time spent T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20250623 01:58:53.072865 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 52544.7us
I20250623 01:58:53.075770 13776 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.077200 15126 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:58:53.074380 15132 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.083628 15331 consensus_queue.cc:260] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.037993 13804 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:53.085613 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:53.086092 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:53.086525 13805 raft_consensus.cc:3053] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:53.087034 13805 raft_consensus.cc:738] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:53.087096 13804 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.088248 13805 consensus_queue.cc:260] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.048280 15149 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.090827 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:53.073459 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.102099 13806 raft_consensus.cc:2408] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250623 01:58:53.096935 14478 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.100888 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:53.111183 13880 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:58:53.092283 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:53.116533 13812 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:53.116712 13697 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 0f12346da5a14097bbfa150cb8d1d907 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica c243715fe12445019c33bbeafbaea635 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:58:53.117338 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:53.117769 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 142698us
I20250623 01:58:53.134161 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.135483 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:53.095454 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.085230 15331 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.139367 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.141750 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:53.091655 15271 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:53.148173 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 52878.3us
I20250623 01:58:53.152419 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:53.160348 13776 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.160699 13809 raft_consensus.cc:2391] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 2: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:53.161376 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:53.161613 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 143853us
I20250623 01:58:53.023555 13709 rpcz_store.cc:276] 0623 01:58:44.813762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:44.814413 (+ 651us) service_pool.cc:225] Handling call
0623 01:58:53.023020 (+8208607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:49.518410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:49.530373 (+ 11963us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:49.530433 (+ 60us) write_op.cc:435] Acquired schema lock
0623 01:58:49.530459 (+ 26us) tablet.cc:660] Decoding operations
0623 01:58:49.559258 (+ 28799us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:49.559370 (+ 112us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:49.559413 (+ 43us) tablet.cc:683] Acquiring locks for 6 operations
0623 01:58:49.560097 (+ 684us) tablet.cc:699] Row locks acquired
0623 01:58:49.560122 (+ 25us) write_op.cc:260] PREPARE: finished
0623 01:58:49.560300 (+ 178us) write_op.cc:270] Start()
0623 01:58:49.560439 (+ 139us) write_op.cc:276] Timestamp: P: 1750643929560243 usec, L: 0
0623 01:58:49.560462 (+ 23us) op_driver.cc:348] REPLICATION: starting
0623 01:58:49.958211 (+397749us) log.cc:844] Serialized 2445 byte log entry
0623 01:58:51.297476 (+1339265us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e0b22e7e 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e36dd04b
0623 01:58:51.810952 (+513476us) op_driver.cc:464] REPLICATION: finished
0623 01:58:52.010135 (+199183us) write_op.cc:301] APPLY: starting
0623 01:58:52.010211 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:52.010369 (+ 158us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:52.010388 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:52.067863 (+ 57475us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:52.079375 (+ 11512us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0623 01:58:52.079426 (+ 51us) write_op.cc:312] APPLY: finished
0623 01:58:52.079659 (+ 233us) log.cc:844] Serialized 52 byte log entry
0623 01:58:52.079863 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:52.079974 (+ 111us) write_op.cc:454] Released schema lock
0623 01:58:52.080022 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:52.080708 (+ 686us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:52.080738 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":562895,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":192621,"apply.run_cpu_time_us":3123,"apply.run_wall_time_us":70567,"num_ops":6,"prepare.queue_time_us":301170,"prepare.run_cpu_time_us":23856,"prepare.run_wall_time_us":1963659,"raft.queue_time_us":148775,"raft.run_cpu_time_us":31590,"raft.run_wall_time_us":255180,"replication_time_us":2181592,"spinlock_wait_cycles":31800704,"thread_start_us":1873394,"threads_started":4,"wal-append.queue_time_us":1427318,"wal-append.run_cpu_time_us":19063,"wal-append.run_wall_time_us":1672905}]]}
I20250623 01:58:53.153378 13805 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.167239 15095 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:53.175472 13880 raft_consensus.cc:1238] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:53.179510 13812 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:53.178258 13960 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:53.095398 13884 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:58:53.184361 13806 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
W20250623 01:58:53.193356 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 46309.2us
I20250623 01:58:53.204315 13810 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
W20250623 01:58:53.206362 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:53.207300 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.195724 15271 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:58:53.210076 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 49594.9us
I20250623 01:58:53.192430 13884 raft_consensus.cc:604] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:53.217571 15235 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250623 01:58:53.236424 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.240365 13811 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
W20250623 01:58:53.241951 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 99003.9us
I20250623 01:58:53.243479 13925 leader_election.cc:400] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:53.191876 15095 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:53.217160 15284 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.023s sys 0.021s
I20250623 01:58:53.244939 15040 consensus_queue.cc:1046] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250623 01:58:53.250536 13809 raft_consensus.cc:1238] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20250623 01:58:53.236650 13884 raft_consensus.cc:612] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:58:53.236414 15235 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:53.217944 15338 consensus_queue.cc:1046] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.090s
I20250623 01:58:53.257012 13811 raft_consensus.cc:1238] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
W20250623 01:58:53.254662 15284 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.890s user 0.025s sys 0.021s
I20250623 01:58:53.264731 15095 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.269392 15338 raft_consensus.cc:3053] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:53.269874 15338 raft_consensus.cc:738] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:53.271171 13925 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.264923 13854 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:53.275817 13810 raft_consensus.cc:1238] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3. Ops: []
W20250623 01:58:53.282001 15366 log.cc:927] Time spent T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250623 01:58:53.284159 15277 consensus_queue.cc:1046] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:53.263381 15235 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.285154 15034 raft_consensus.cc:3053] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:53.285614 15034 raft_consensus.cc:738] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:53.286556 15034 consensus_queue.cc:260] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.287436 15252 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:53.287837 15034 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.287981 15252 raft_consensus.cc:695] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:53.289155 15252 consensus_queue.cc:237] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.295873 13776 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.269423 15095 consensus_queue.cc:1046] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
W20250623 01:58:53.296873 15316 consensus_meta.cc:287] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.648s user 0.006s sys 0.011s
I20250623 01:58:53.297677 13886 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
W20250623 01:58:53.298786 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 57022.4us
I20250623 01:58:53.297196 13962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1083) took 3908 ms. Trace:
I20250623 01:58:53.301421 13808 raft_consensus.cc:1238] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20250623 01:58:53.272727 15040 consensus_queue.cc:1230] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250623 01:58:53.302737 13885 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250623 01:58:53.297907 15277 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250623 01:58:53.271052 15338 consensus_queue.cc:260] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.307612 14926 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:53.311144 13925 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:53.314872 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.300052 13962 rpcz_store.cc:276] 0623 01:58:49.388350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:52.082317 (+2693967us) service_pool.cc:225] Handling call
0623 01:58:52.082592 (+ 275us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:52.085228 (+ 2636us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:52.085257 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:52.085624 (+ 367us) log.cc:844] Serialized 26 byte log entry
0623 01:58:52.797052 (+711428us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b18000046c0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b22bc5 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e36dd04b 00007f41e36e2b53 00007f41e36e2f01 00007f41e3a71ccf
0623 01:58:52.975681 (+178629us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:52.979002 (+ 3321us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:52.987360 (+ 8358us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:53.294703 (+307343us) raft_consensus.cc:1683] finished
0623 01:58:53.294719 (+ 16us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:53.297015 (+ 2296us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2278,"raft.queue_time_us":5293,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":8377216,"thread_start_us":104232,"threads_started":1,"wal-append.queue_time_us":1201518}
I20250623 01:58:53.324627 15106 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:53.324498 15040 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:53.325289 15106 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.329062 13810 raft_consensus.cc:1238] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250623 01:58:53.327947 13808 raft_consensus.cc:1238] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:53.331372 13810 raft_consensus.cc:1238] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:53.331431 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.315958 13804 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
W20250623 01:58:53.335002 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 79685.3us
I20250623 01:58:53.341578 15040 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:53.345260 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.346354 13926 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:53.348063 15040 raft_consensus.cc:2747] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:53.348233 15367 log.cc:927] Time spent T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.159s user 0.004s sys 0.000s
I20250623 01:58:53.326994 15284 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.338511 15230 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
W20250623 01:58:53.366413 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 62019.7us
I20250623 01:58:53.362928 13926 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.367518 13925 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.367700 15338 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:53.376909 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.359185 15350 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20250623 01:58:53.377575 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.367861 15230 raft_consensus.cc:695] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:53.371484 15106 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.386782 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:53.390465 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.387615 13962 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:58:53.376776 15277 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:53.392709 15040 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
W20250623 01:58:53.399820 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 46641.5us
I20250623 01:58:53.400756 13807 raft_consensus.cc:1238] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:53.399336 15040 raft_consensus.cc:695] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:53.416529 13776 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
W20250623 01:58:53.413350 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 57300.5us
I20250623 01:58:53.418622 13810 raft_consensus.cc:1238] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:53.400990 14481 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:53.424118 13805 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:53.425501 14481 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:53.426436 15149 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:58:53.426832 15149 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:53.427201 15149 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:53.445274 13810 raft_consensus.cc:1238] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:53.410578 15235 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.448760 13854 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:53.443559 15106 raft_consensus.cc:2802] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:53.443750 15284 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:53.453877 15106 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:53.461638 13811 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:53.461637 13807 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250623 01:58:53.464138 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 53581.1us
I20250623 01:58:53.464567 13812 raft_consensus.cc:1273] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:53.465189 13776 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.465935 15235 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.454546 15284 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:53.467602 15271 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:53.467900 15284 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.468161 15271 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:53.468677 15271 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.462383 13805 raft_consensus.cc:1238] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:53.460847 14508 consensus_queue.cc:237] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.473412 15106 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.474133 15040 consensus_queue.cc:237] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.475809 13776 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:53.476822 15126 raft_consensus.cc:2802] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:53.477210 15126 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:53.477599 15126 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.477888 13854 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:53.480484 13854 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.462086 13926 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.455554 15073 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.481906 15095 raft_consensus.cc:2762] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250623 01:58:53.459432 15230 consensus_queue.cc:237] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.486687 15149 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.488766 15149 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.490063 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:53.491591 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 222257us
I20250623 01:58:53.498860 15095 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:53.499044 13852 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:53.499833 15316 raft_consensus.cc:3053] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:53.500280 15316 raft_consensus.cc:738] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.515180 13813 raft_consensus.cc:3053] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:53.515636 13813 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:53.518412 13852 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.494251 15132 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.507159 15095 raft_consensus.cc:695] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:58:53.521275 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 73734.9us
I20250623 01:58:53.522064 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:53.506992 15331 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:53.523478 13813 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.524415 14967 raft_consensus.cc:3053] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:53.521400 13852 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:58:53.533118 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 57440.3us
I20250623 01:58:53.539420 13806 raft_consensus.cc:1238] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:53.537922 14926 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:53.547346 15381 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:53.539459 15316 consensus_queue.cc:260] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.550365 15316 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:53.549657 15073 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:53.553395 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:53.554092 13954 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:53.553791 15095 consensus_queue.cc:237] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.533427 14967 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:53.564199 14926 raft_consensus.cc:695] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.565150 14926 consensus_queue.cc:237] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.575480 13813 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.577464 14967 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.578253 15132 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.578473 14967 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:53.578943 14948 raft_consensus.cc:3053] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:53.579345 14948 raft_consensus.cc:738] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:53.579568 15271 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.580230 14948 consensus_queue.cc:260] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.580557 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.586925 15316 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:53.587457 15316 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.589259 15316 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.597043 13954 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:58:53.599130 13852 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:58:53.599927 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.601651 15126 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.604209 15126 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:53.605720 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
W20250623 01:58:53.609732 15379 log.cc:927] Time spent T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20250623 01:58:53.603778 15284 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.613688 15331 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:53.614166 15230 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:53.614342 15132 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:53.614696 15230 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:53.615989 13776 leader_election.cc:400] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:53.616318 13776 leader_election.cc:403] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:53.616706 15230 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:53.617056 15371 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3.
I20250623 01:58:53.617441 13776 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.618248 15230 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:53.618832 15230 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.620663 13776 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.621423 15230 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:53.621712 15271 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.621615 15383 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:53.622378 15271 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:53.626060 15106 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.629122 14481 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:53.630462 15149 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:58:53.630970 15149 raft_consensus.cc:695] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:53.631819 15149 consensus_queue.cc:237] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.633615 15284 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.633822 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.634434 13884 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:53.634891 15390 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20250623 01:58:53.641983 13852 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.643049 15235 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:53.643098 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.643678 15235 raft_consensus.cc:695] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.644742 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.644593 15235 consensus_queue.cc:237] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.645507 15316 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:53.645607 13805 raft_consensus.cc:2366] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:53.646513 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.644831 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.648017 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:53.648859 15386 consensus_queue.cc:1046] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:53.649449 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:53.649961 15385 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250623 01:58:53.650238 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:53.650863 13806 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:53.653293 15382 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20250623 01:58:53.654084 14948 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.657353 15073 consensus_queue.cc:1046] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:53.662573 13813 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250623 01:58:53.663108 13813 pending_rounds.cc:85] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 1
I20250623 01:58:53.663476 13813 pending_rounds.cc:107] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250623 01:58:53.664108 13813 raft_consensus.cc:2887] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:53.666608 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.667176 13807 raft_consensus.cc:2374] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3. Re-sending same reply.
I20250623 01:58:53.650281 15316 raft_consensus.cc:3053] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:53.670380 13808 raft_consensus.cc:1238] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:53.669754 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:53.670465 15316 raft_consensus.cc:738] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.672545 15316 consensus_queue.cc:260] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:53.677299 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 96872.2us
I20250623 01:58:53.680027 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:53.686048 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 65501us
W20250623 01:58:53.691980 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:53.684049 15242 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20250623 01:58:53.699186 15358 consensus_queue.cc:1046] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:58:53.700407 15358 raft_consensus.cc:3053] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:53.700784 15358 raft_consensus.cc:738] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
W20250623 01:58:53.702708 15388 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20250623 01:58:53.695039 15073 raft_consensus.cc:3053] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:53.705703 15073 raft_consensus.cc:738] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.706549 15073 consensus_queue.cc:260] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.708828 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:53.700831 15316 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:53.709431 13954 raft_consensus.cc:3058] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.712680 13806 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:53.714932 13926 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.717465 13954 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:53.685387 15106 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:53.732128 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 199484us
W20250623 01:58:53.722748 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 65843.9us
I20250623 01:58:53.718122 15230 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:58:53.744009 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 63133.3us
I20250623 01:58:53.720049 13884 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
W20250623 01:58:53.752547 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.745991 15371 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.763468 13926 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:53.768642 13926 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:53.750105 15358 consensus_queue.cc:260] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.771894 15386 consensus_queue.cc:1046] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:58:53.772789 15358 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.742445 14481 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:53.776216 15106 raft_consensus.cc:2802] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:53.776674 15106 raft_consensus.cc:695] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:53.769276 15230 raft_consensus.cc:2762] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250623 01:58:53.779034 15316 raft_consensus.cc:3053] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:53.779435 15316 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.779935 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:53.780335 15316 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.781095 13886 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250623 01:58:53.781513 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.783037 13886 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:58:53.783564 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:53.784364 13884 raft_consensus.cc:2391] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:53.784930 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:53.785615 13886 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:53.786644 15381 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:53.787006 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.794085 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.794706 13926 leader_election.cc:400] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:53.795975 13926 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.796943 14967 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:53.797377 14967 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:53.797504 15230 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.797770 14967 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.764614 15073 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.801623 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:53.802520 15386 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:58:53.802870 15386 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:53.803217 15386 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:53.806515 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.807268 13886 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:53.808058 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:53.777608 15106 consensus_queue.cc:237] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.810583 15386 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.812408 15386 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.813711 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:58:53.816781 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 55065.2us
I20250623 01:58:53.817972 13926 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.821955 13926 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.822703 15034 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.823297 15034 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.823951 15316 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.823199 13926 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.829382 13854 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:53.830933 15395 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.831373 13854 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:53.832620 13854 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:53.832828 15395 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:58:53.833422 14926 raft_consensus.cc:2802] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:53.833789 14926 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:53.834180 14926 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:53.829272 15316 consensus_queue.cc:1046] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20250623 01:58:53.818316 15381 consensus_queue.cc:1046] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250623 01:58:53.836429 15095 raft_consensus.cc:3053] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:53.836612 15397 consensus_queue.cc:1046] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:53.836966 15095 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.836498 15381 raft_consensus.cc:3053] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:53.836449 15034 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:53.837623 15381 raft_consensus.cc:738] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.836751 15386 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:53.837875 15095 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.839042 15095 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.838637 15381 consensus_queue.cc:260] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.840297 15381 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:53.853606 13926 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.854907 13926 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.855727 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.855780 13961 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:53.859318 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:53.863435 13880 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:53.863829 13804 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:58:53.864743 13804 raft_consensus.cc:604] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:53.865185 13804 raft_consensus.cc:612] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:58:53.865415 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.865803 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.866324 13883 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:53.866636 14926 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.867422 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
W20250623 01:58:53.869444 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.870851 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:53.871480 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:53.872202 13886 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:53.818126 14967 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:53.870798 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.873220 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.874691 15230 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:53.875924 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:53.879951 14967 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:53.880666 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:53.881278 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:53.888720 13962 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.887516 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:53.881232 15034 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:53.889744 13955 raft_consensus.cc:2366] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:58:53.892194 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:53.895722 13776 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.896373 15396 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:53.897017 15396 consensus_queue.cc:1230] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250623 01:58:53.897454 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:53.897796 13959 raft_consensus.cc:1238] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:53.898437 15235 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:53.898610 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:53.899190 13960 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:53.899094 15235 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.900642 13882 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:53.898345 15371 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:53.901131 15235 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:53.903728 13776 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.905138 15242 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:53.905550 15242 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:53.905890 15242 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:53.909688 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:53.910295 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 55503.4us
I20250623 01:58:53.904804 13776 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.913447 13961 raft_consensus.cc:2408] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250623 01:58:53.913918 13960 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:58:53.914022 13776 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.915409 13854 leader_election.cc:400] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:58:53.915709 13854 leader_election.cc:403] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250623 01:58:53.868685 14926 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:53.917273 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:53.918743 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:53.920725 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:53.917466 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 82185.9us
I20250623 01:58:53.922078 13955 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:53.921905 13880 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:53.923715 13776 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.913424 15284 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:53.916865 15396 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:53.890429 15073 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250623 01:58:53.926651 15271 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:53.917367 15397 raft_consensus.cc:3058] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:53.921195 15126 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:53.926888 13882 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20250623 01:58:53.927449 13711 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:53.926342 13885 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:53.930658 13883 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:53.925590 13884 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:53.925549 13854 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.931141 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 61) took 4741 ms. Trace:
I20250623 01:58:53.935365 15284 raft_consensus.cc:695] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:53.922552 15132 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20250623 01:58:53.936316 15284 consensus_queue.cc:237] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.940615 13711 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:53.929570 15396 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:53.944343 15397 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:58:53.944501 15396 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:53.934614 15405 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:58:53.950433 13962 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:53.934230 13708 rpcz_store.cc:276] 0623 01:58:49.184727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:49.184876 (+ 149us) service_pool.cc:225] Handling call
0623 01:58:53.925822 (+4740946us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:53.291832 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:53.292737 (+ 905us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:53.292794 (+ 57us) write_op.cc:435] Acquired schema lock
0623 01:58:53.292813 (+ 19us) tablet.cc:660] Decoding operations
0623 01:58:53.293403 (+ 590us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:53.293498 (+ 95us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:53.293531 (+ 33us) tablet.cc:683] Acquiring locks for 2 operations
0623 01:58:53.294027 (+ 496us) tablet.cc:699] Row locks acquired
0623 01:58:53.294049 (+ 22us) write_op.cc:260] PREPARE: finished
0623 01:58:53.305898 (+ 11849us) write_op.cc:270] Start()
0623 01:58:53.314837 (+ 8939us) write_op.cc:276] Timestamp: P: 1750643933302645 usec, L: 0
0623 01:58:53.318776 (+ 3939us) op_driver.cc:348] REPLICATION: starting
0623 01:58:53.378381 (+ 59605us) log.cc:844] Serialized 967 byte log entry
0623 01:58:53.648596 (+270215us) op_driver.cc:464] REPLICATION: finished
0623 01:58:53.762255 (+113659us) spinlock_profiling.cc:243] Waited 9.42 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e0b22e7e 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e74c1180
0623 01:58:53.889933 (+127678us) write_op.cc:301] APPLY: starting
0623 01:58:53.890058 (+ 125us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:53.890948 (+ 890us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:53.890988 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:53.895550 (+ 4562us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:53.895839 (+ 289us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 01:58:53.895901 (+ 62us) write_op.cc:312] APPLY: finished
0623 01:58:53.896168 (+ 267us) log.cc:844] Serialized 28 byte log entry
0623 01:58:53.896346 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:53.896438 (+ 92us) write_op.cc:454] Released schema lock
0623 01:58:53.896503 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:53.896835 (+ 332us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:53.896864 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":32387,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":241017,"apply.run_cpu_time_us":2269,"apply.run_wall_time_us":6903,"num_ops":2,"prepare.queue_time_us":414068,"prepare.run_cpu_time_us":19421,"prepare.run_wall_time_us":182981,"raft.queue_time_us":99402,"raft.run_cpu_time_us":26435,"raft.run_wall_time_us":170554,"replication_time_us":329660,"spinlock_wait_cycles":21671296,"thread_start_us":513446,"threads_started":4,"wal-append.queue_time_us":114162}]]}
I20250623 01:58:53.951352 15271 raft_consensus.cc:695] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:53.950026 13711 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:53.917224 15235 consensus_queue.cc:1046] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20250623 01:58:53.952384 15271 consensus_queue.cc:237] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.952674 13711 catalog_manager.cc:5582] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 3, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:53.956198 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:53.957195 13926 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.954386 13711 catalog_manager.cc:5582] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 2, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:53.959642 15095 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:58:53.963876 15381 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:53.964149 15095 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:53.964474 15381 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.960423 15405 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.964543 15242 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.965461 15405 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.966821 15242 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:53.969619 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.970973 15272 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:53.972460 15395 consensus_queue.cc:1046] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:53.965461 15381 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:53.980258 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:53.964591 15095 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:53.987895 15316 raft_consensus.cc:3053] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:53.988453 15316 raft_consensus.cc:738] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:53.989502 15316 consensus_queue.cc:260] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.994455 15132 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:53.994979 15132 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:53.995792 13926 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:54.002274 13962 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:54.003170 15272 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:54.003724 15272 raft_consensus.cc:695] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:54.004563 15272 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.007366 15106 consensus_queue.cc:1046] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:58:54.008132 15106 raft_consensus.cc:3053] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:54.008414 15106 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:54.009214 15106 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:54.010800 15106 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:54.002971 13926 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:53.996941 15132 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:54.046871 15040 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:54.000720 15396 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.062264 15278 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.064872 14967 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:54.061445 15034 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:54.049880 15230 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:58:54.122375 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 119509us
I20250623 01:58:54.123140 13926 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.125926 15230 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:54.119300 15095 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.127040 13926 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:54.129420 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:54.129940 13961 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:58:54.130944 15331 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:54.131455 15331 raft_consensus.cc:695] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:54.124575 15034 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:54.122313 15278 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:54.132339 15331 consensus_queue.cc:237] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.136034 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:54.160068 14507 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:54.161985 13927 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:54.138314 15040 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:54.138727 15034 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:54.152473 13961 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:58:54.154580 13955 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:58:54.159183 15132 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.169571 15272 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:54.183552 15132 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.184139 15272 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:54.184254 15396 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:54.184562 15272 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:54.122758 14967 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:54.186404 13854 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:54.185848 15132 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:54.180456 13708 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 3 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:54.153484 15095 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:54.205580 13708 catalog_manager.cc:5582] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 4 to 6, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 6 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:54.162868 15271 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.220568 13710 catalog_manager.cc:5582] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 2, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
W20250623 01:58:54.231181 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:54.212481 15397 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
W20250623 01:58:54.232334 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 309327us
I20250623 01:58:54.217348 13708 catalog_manager.cc:5582] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 2 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 4 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:54.235272 13776 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:54.232338 13710 catalog_manager.cc:5582] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:54.238024 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 172) took 8761 ms. Trace:
I20250623 01:58:54.237591 13710 catalog_manager.cc:5582] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
W20250623 01:58:54.243234 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 116251us
W20250623 01:58:54.246824 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:54.242422 13697 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 0f12346da5a14097bbfa150cb8d1d907 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica c243715fe12445019c33bbeafbaea635 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:54.248353 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.244690 13710 catalog_manager.cc:5582] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:54.252611 13710 catalog_manager.cc:5582] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:54.254385 13776 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:54.241297 15396 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:54.238591 13711 rpcz_store.cc:276] 0623 01:58:45.470682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:45.470808 (+ 126us) service_pool.cc:225] Handling call
0623 01:58:45.754217 (+283409us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b54003868d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e97bafac 00007f41e98b37ee 00007f41e7dfd624 00007f41e7dfd5b2 00007f41e7dfd4e1 00007f41e7dfd45c 00007f41e7dfc712 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:54.232096 (+8477879us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:54.144081 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:54.144642 (+ 561us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:54.144682 (+ 40us) write_op.cc:435] Acquired schema lock
0623 01:58:54.144700 (+ 18us) tablet.cc:660] Decoding operations
0623 01:58:54.145416 (+ 716us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:54.145475 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:54.145504 (+ 29us) tablet.cc:683] Acquiring locks for 5 operations
0623 01:58:54.146062 (+ 558us) tablet.cc:699] Row locks acquired
0623 01:58:54.146082 (+ 20us) write_op.cc:260] PREPARE: finished
0623 01:58:54.146217 (+ 135us) write_op.cc:270] Start()
0623 01:58:54.146331 (+ 114us) write_op.cc:276] Timestamp: P: 1750643934146171 usec, L: 0
0623 01:58:54.146348 (+ 17us) op_driver.cc:348] REPLICATION: starting
0623 01:58:54.147092 (+ 744us) log.cc:844] Serialized 2063 byte log entry
0623 01:58:54.148869 (+ 1777us) op_driver.cc:464] REPLICATION: finished
0623 01:58:54.149457 (+ 588us) write_op.cc:301] APPLY: starting
0623 01:58:54.149553 (+ 96us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:54.149737 (+ 184us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:54.149836 (+ 99us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:54.156507 (+ 6671us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:54.156670 (+ 163us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0623 01:58:54.156709 (+ 39us) write_op.cc:312] APPLY: finished
0623 01:58:54.156934 (+ 225us) log.cc:844] Serialized 46 byte log entry
0623 01:58:54.157099 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:54.157208 (+ 109us) write_op.cc:454] Released schema lock
0623 01:58:54.157245 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:54.157582 (+ 337us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:54.157607 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"spinlock_wait_cycles":134617856,"child_traces":[["op",{"apply.queue_time_us":307,"apply.run_cpu_time_us":2290,"apply.run_wall_time_us":8118,"num_ops":5,"prepare.queue_time_us":163389,"prepare.run_cpu_time_us":3319,"prepare.run_wall_time_us":4712,"replication_time_us":2359,"thread_start_us":111323,"threads_started":1}]]}
I20250623 01:58:54.240998 15132 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250623 01:58:54.230515 15271 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:54.242424 15242 consensus_queue.cc:1046] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20250623 01:58:54.263182 15126 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
W20250623 01:58:54.286159 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:54.243872 15149 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20250623 01:58:54.298719 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.299672 13880 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:54.300681 13883 raft_consensus.cc:1238] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:54.301043 14967 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:54.303757 13710 catalog_manager.cc:5582] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:54.259160 15040 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:58:54.289327 13954 raft_consensus.cc:1273] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:54.312525 13955 raft_consensus.cc:1273] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250623 01:58:54.312935 13955 pending_rounds.cc:85] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 1
I20250623 01:58:54.313179 13955 pending_rounds.cc:107] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250623 01:58:54.313457 13955 raft_consensus.cc:2887] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:54.311276 13776 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:54.285719 13886 raft_consensus.cc:1238] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:54.322329 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:54.323156 13882 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:54.325465 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:54.326099 13881 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:54.326962 15395 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250623 01:58:54.328483 15417 log.cc:927] Time spent T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20250623 01:58:54.329473 15419 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.316918 14967 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
I20250623 01:58:54.329967 15419 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.331032 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.332237 15419 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:54.293531 15316 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:54.335216 13886 raft_consensus.cc:1273] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:58:54.305392 13710 catalog_manager.cc:5582] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 3 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:54.339450 13710 catalog_manager.cc:5582] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 2, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:54.341275 13710 catalog_manager.cc:5582] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:54.359405 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:54.359730 15421 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250623 01:58:54.373394 13710 catalog_manager.cc:5582] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 3 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:54.360131 13804 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:54.324159 15278 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:54.325465 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:54.378391 13926 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:54.378638 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 140897us
W20250623 01:58:54.386572 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:54.341881 15126 raft_consensus.cc:695] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:54.397719 13882 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:54.365828 15396 raft_consensus.cc:695] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:54.354457 15397 raft_consensus.cc:695] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:54.404840 14967 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:54.414820 15271 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:54.417290 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:54.419404 15126 consensus_queue.cc:237] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.400735 14507 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:54.399682 15132 consensus_queue.cc:1230] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250623 01:58:54.426398 13927 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:54.337893 15418 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20250623 01:58:54.427330 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:54.427886 13962 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250623 01:58:54.427937 13960 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:54.338996 15235 consensus_queue.cc:1035] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20250623 01:58:54.436830 13886 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250623 01:58:54.375115 13710 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:54.442123 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.443723 13710 catalog_manager.cc:5582] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 2 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:54.446058 13710 catalog_manager.cc:5582] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:54.447420 13710 catalog_manager.cc:5582] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:54.448282 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.449244 13880 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:54.448747 13710 catalog_manager.cc:5582] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 2 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
W20250623 01:58:54.451115 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 72782.7us
I20250623 01:58:54.450680 13710 catalog_manager.cc:5582] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:54.379839 15331 consensus_queue.cc:1046] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20250623 01:58:54.459266 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.459853 13880 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:54.375283 15237 consensus_queue.cc:1035] T 433978dce7d94bcb8968bc09583357c4 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20250623 01:58:54.456200 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:54.462653 13926 leader_election.cc:400] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3.
I20250623 01:58:54.429702 15230 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.429447 15381 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20250623 01:58:54.464308 13886 raft_consensus.cc:1238] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250623 01:58:54.444623 15271 raft_consensus.cc:3053] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:54.445854 15040 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:54.533725 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:58:54.535336 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:54.429561 14481 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:58:54.547201 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 236043us
I20250623 01:58:54.548312 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.551805 13886 raft_consensus.cc:2391] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:54.553030 15230 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:54.553071 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 101103us
I20250623 01:58:54.556553 13926 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.466676 15428 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:54.565443 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:54.565972 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:54.561954 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.566772 13807 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:54.537986 15271 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:54.567240 13880 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:54.465831 15106 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.093s
I20250623 01:58:54.569473 15434 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:54.569705 15338 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:54.546394 15040 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:54.590224 13807 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250623 01:58:54.559160 13779 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.525s (SENT) [suppressed 20 similar messages]
I20250623 01:58:54.561384 13925 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.594491 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:54.608273 13813 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:54.594858 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:54.597949 13926 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.600988 13879 raft_consensus.cc:1273] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:54.601887 13881 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:58:54.602011 13809 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:54.615945 13887 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:54.604703 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:54.610486 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 49166us
I20250623 01:58:54.567746 15149 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:54.568048 15428 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:54.622289 15428 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:54.614636 15427 consensus_queue.cc:1046] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:54.616612 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:54.562016 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 132551us
I20250623 01:58:54.621518 15149 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:54.621670 15429 raft_consensus.cc:3053] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:58:54.630259 13810 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:54.613476 15040 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:54.618809 14967 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:58:54.618394 15230 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:54.649912 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 58217.8us
I20250623 01:58:54.652905 13776 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:54.655627 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 57782.9us
I20250623 01:58:54.656543 13926 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.659494 15427 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:54.589509 15396 consensus_queue.cc:237] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.660076 15427 raft_consensus.cc:695] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:54.660396 15230 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:58:54.637938 15149 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:54.637658 15429 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:54.590541 15397 consensus_queue.cc:237] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.640969 15419 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:54.669085 13926 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.653893 15371 raft_consensus.cc:3053] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:54.605792 15434 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.679949 14967 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.690294 14967 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.692260 14967 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:54.664847 15237 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.029s
I20250623 01:58:54.694811 15430 consensus_queue.cc:1046] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20250623 01:58:54.696833 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.698030 15338 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:54.698266 13926 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:54.698868 15338 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:54.699613 15338 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:54.700225 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:54.700323 14926 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:54.700874 13960 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 1.
I20250623 01:58:54.701090 14926 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:54.701905 14967 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.702384 14967 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:54.703058 15272 consensus_meta.cc:287] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.518s user 0.008s sys 0.003s
I20250623 01:58:54.703423 15272 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:54.704208 14926 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:54.704241 14967 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:54.705092 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.634264 15271 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.686127 15371 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:54.712098 13810 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
W20250623 01:58:54.713279 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 81753.3us
I20250623 01:58:54.726354 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:54.706522 15271 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:54.732933 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:54.733747 14508 raft_consensus.cc:2391] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 4: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:54.720371 15429 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.736634 15429 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:54.737382 15416 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:54.737924 15416 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.744451 13852 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:54.744951 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:54.745853 15426 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.746449 15426 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.747089 15444 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:54.747531 15444 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:54.747901 15444 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:54.749718 15423 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.705296 15272 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:54.750247 15423 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.660971 15427 consensus_queue.cc:237] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.752285 15423 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:54.753798 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 98088.4us
I20250623 01:58:54.755851 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:54.756632 13962 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
W20250623 01:58:54.757483 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:54.753482 15380 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.181s user 0.003s sys 0.000s
I20250623 01:58:54.717814 15230 consensus_queue.cc:1046] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250623 01:58:54.719923 14967 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20250623 01:58:54.783324 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 224200us
I20250623 01:58:54.785543 13808 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:58:54.794380 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:54.795221 13959 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
W20250623 01:58:54.803203 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:54.803083 15426 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:54.810784 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:54.784092 15423 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20250623 01:58:54.813568 13776 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:54.707530 15371 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:54.776418 15395 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250623 01:58:54.827363 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:54.832867 13810 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20250623 01:58:54.851117 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 96635.6us
W20250623 01:58:54.853003 13779 leader_election.cc:336] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.594s (SENT)
I20250623 01:58:54.853514 13779 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:54.865541 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 89307.4us
I20250623 01:58:54.868144 13854 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:54.849906 15416 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:54.872566 13955 raft_consensus.cc:1238] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:54.861631 15371 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:54.882902 15395 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:58:54.883299 15395 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:54.883602 15395 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:54.857812 15242 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:54.884331 15040 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:54.885751 13961 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
W20250623 01:58:54.888769 15316 consensus_meta.cc:287] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.537s user 0.006s sys 0.001s
I20250623 01:58:54.873905 15106 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.891062 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:54.892009 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 129068us
I20250623 01:58:54.927783 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:54.873571 15434 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:54.937355 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 84402.7us
I20250623 01:58:54.939911 13812 raft_consensus.cc:2391] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:54.960639 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:54.964372 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 44726.9us
I20250623 01:58:54.965788 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:54.966526 13810 raft_consensus.cc:2391] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 4: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:58:54.966406 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:54.908476 15396 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:54.890244 15449 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:54.909540 13805 raft_consensus.cc:1273] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:54.971854 13883 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:54.972823 13887 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
W20250623 01:58:55.006142 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 138043us
I20250623 01:58:54.923905 15446 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.010218 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:54.932525 15242 raft_consensus.cc:695] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:55.029953 13885 raft_consensus.cc:2391] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:58:55.031339 13779 leader_election.cc:336] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.825s (SENT)
I20250623 01:58:55.050076 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.050770 13887 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:55.052126 13813 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20250623 01:58:55.052611 13813 pending_rounds.cc:85] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 2
I20250623 01:58:55.053009 13813 pending_rounds.cc:107] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250623 01:58:55.053287 13813 raft_consensus.cc:2887] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:54.928771 15106 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.057354 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:55.011202 15237 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.050469 13962 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7560a94e265b433dbc427ef13c120669"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37424
I20250623 01:58:55.072088 13962 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:58:55.072485 13962 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:58:55.076351 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 60706.4us
I20250623 01:58:55.041535 15456 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:55.116503 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.122365 13956 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:55.117502 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:55.071154 15380 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.252s user 0.002s sys 0.000s
I20250623 01:58:55.044292 15434 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:55.021937 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 48353.9us
I20250623 01:58:55.077247 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:55.135792 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.007179 15428 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.068609 15396 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:55.137347 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 60272.7us
I20250623 01:58:55.046927 15230 consensus_queue.cc:1046] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.084s
W20250623 01:58:55.141561 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 98522.9us
I20250623 01:58:55.110494 15429 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:55.110219 15132 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:55.150302 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 129664us
I20250623 01:58:55.143872 13854 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:55.151366 13852 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:55.022150 15449 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.132292 15423 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:55.048180 15446 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:55.163380 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 154131us
I20250623 01:58:55.174374 13962 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:55.178699 13779 leader_election.cc:336] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.858s (SENT)
W20250623 01:58:55.164225 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.170361 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.137971 15426 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.179798 13956 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:55.157506 13806 raft_consensus.cc:2391] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 5: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:55.108304 15237 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.146246 13804 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:55.187958 13779 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:55.188119 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.155422 15040 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:55.187585 15447 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.181237 13962 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:55.151311 15396 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.156123 15434 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.164680 15132 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.234441 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.140120 15456 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.155261 15429 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.199420 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
W20250623 01:58:55.244626 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 65959.6us
W20250623 01:58:55.227517 13879 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.003s sys 0.015s
I20250623 01:58:55.259580 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:55.193192 15423 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.274464 15469 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:55.275211 15434 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.275794 15423 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:55.274188 15371 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:55.245813 15433 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.194s
W20250623 01:58:55.276618 15460 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20250623 01:58:55.227152 15427 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:55.279240 15427 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:55.281455 15443 log.cc:927] Time spent T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.317s user 0.000s sys 0.001s
I20250623 01:58:55.281553 15427 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.268258 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.191951 15426 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.283252 15467 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.283903 15467 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.285219 15426 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:55.272871 15459 log.cc:927] Time spent T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20250623 01:58:55.273859 15429 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:55.273914 15451 log.cc:927] Time spent T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20250623 01:58:55.190151 15242 consensus_queue.cc:237] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.287135 15467 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:55.288118 15458 log.cc:927] Time spent T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.269s user 0.001s sys 0.000s
I20250623 01:58:55.288334 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:58:55.289086 15439 log.cc:927] Time spent T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.310s user 0.001s sys 0.001s
I20250623 01:58:55.289269 14509 raft_consensus.cc:2408] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20250623 01:58:55.282876 15441 log.cc:927] Time spent T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.334s user 0.001s sys 0.000s
I20250623 01:58:55.295872 13962 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:55.297353 15456 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:55.299190 15338 raft_consensus.cc:3053] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:55.230477 15284 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:55.305797 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.306516 13962 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
W20250623 01:58:55.261140 13879 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.003s sys 0.016s
W20250623 01:58:55.315608 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:55.243535 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 282001us
I20250623 01:58:55.320679 15426 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.320827 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:55.321244 15476 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:55.322348 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:55.305415 15106 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:55.329988 15426 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.335556 15426 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.336380 14512 raft_consensus.cc:2408] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250623 01:58:55.337771 15449 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.338033 15237 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.299232 15132 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.299584 15338 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:55.341002 15338 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.225960 15447 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.341990 15338 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:55.342420 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:55.329957 14509 raft_consensus.cc:1811] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Handling vote request from an unknown peer 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:55.343307 14509 raft_consensus.cc:2408] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20250623 01:58:55.343456 15447 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.331956 15476 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.345990 15476 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:55.332666 15271 consensus_meta.cc:287] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.591s user 0.009s sys 0.000s
I20250623 01:58:55.331674 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.352809 13956 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
W20250623 01:58:55.336323 15444 consensus_meta.cc:287] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.588s user 0.009s sys 0.000s
I20250623 01:58:55.330107 15428 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.367219 15237 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.367765 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:55.367748 15237 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.368518 14509 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:55.323262 15396 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.370337 15237 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.375738 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.376412 13959 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:55.381795 13811 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:55.393797 15034 raft_consensus.cc:3053] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:55.394336 15034 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:55.395346 15034 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.378926 15272 raft_consensus.cc:3053] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:55.400254 15272 raft_consensus.cc:738] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:55.401111 15272 consensus_queue.cc:260] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.402311 15448 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.402791 15448 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.404793 15448 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.307327 15284 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.407866 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.409817 13953 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:55.426080 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.427153 13961 raft_consensus.cc:2408] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:55.418331 15430 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.375137 15444 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.433928 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.434645 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.442142 13955 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:55.446404 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.452634 13958 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:58:55.452994 13962 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:55.450031 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.458289 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.459093 13955 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.463836 13883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 608) took 1035 ms. Trace:
I20250623 01:58:55.464336 13883 rpcz_store.cc:276] 0623 01:58:54.428616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:54.429855 (+ 1239us) service_pool.cc:225] Handling call
0623 01:58:55.463639 (+1033784us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b540029c2d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:55.463772 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":982844160}
I20250623 01:58:55.467420 13886 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:55.457898 15034 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:55.470350 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:55.470981 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.471153 13962 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:55.474352 13958 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: []
I20250623 01:58:55.486603 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:55.487345 13813 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:55.488437 14481 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:55.493518 14481 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:55.480398 13953 raft_consensus.cc:2391] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:55.495651 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.496858 13955 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
W20250623 01:58:55.481453 15149 consensus_meta.cc:287] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5: Time spent flushing consensus metadata: real 0.810s user 0.010s sys 0.001s
I20250623 01:58:55.497012 13958 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:55.450958 15430 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.473093 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:55.487802 15446 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.507819 15463 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.502708 15149 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.515136 14508 raft_consensus.cc:2366] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:55.516542 15463 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.520804 15430 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.502429 15358 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:55.521944 15358 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:55.522351 15358 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.523183 14507 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:55.501403 15433 raft_consensus.cc:2802] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:55.526044 15433 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:58:55.526531 15395 consensus_meta.cc:287] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.643s user 0.009s sys 0.001s
I20250623 01:58:55.527019 15395 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.529223 15440 raft_consensus.cc:3053] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:55.529798 15440 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:55.530655 15395 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.530586 15440 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:55.531628 15440 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.533480 13888 raft_consensus.cc:2391] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:55.527456 15040 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:55.533830 15463 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:55.534246 15040 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:55.534672 15485 log.cc:927] Time spent T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20250623 01:58:55.536223 15040 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:55.526469 15433 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.538399 15272 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.538681 15444 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.540020 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.540185 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1173) took 1245 ms. Trace:
I20250623 01:58:55.541643 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.542204 13958 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:55.547857 13927 leader_election.cc:400] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Vote denied by peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393) with higher term. Message: Invalid argument: T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:55.548283 13927 leader_election.cc:403] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250623 01:58:55.549085 13927 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:58:55.549912 15040 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.552359 13955 raft_consensus.cc:1273] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:55.553311 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.553972 13955 raft_consensus.cc:2391] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:55.554270 15463 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:55.554634 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.554967 15463 raft_consensus.cc:695] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:55.555754 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.556321 13955 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:55.556206 15463 consensus_queue.cc:237] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.557237 13955 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:55.563596 15040 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393) with higher term. Message: Invalid argument: T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:55.540879 13957 raft_consensus.cc:2408] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:58:55.540588 13884 rpcz_store.cc:276] 0623 01:58:54.295083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:54.313172 (+ 18089us) service_pool.cc:225] Handling call
0623 01:58:55.540005 (+1226833us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54002805d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:55.540129 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2694148992}
I20250623 01:58:55.507534 13805 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:55.566018 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:55.566565 13806 raft_consensus.cc:3058] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.571139 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.571592 15316 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.571681 13958 raft_consensus.cc:2374] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3. Re-sending same reply.
I20250623 01:58:55.572163 15316 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.547874 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.565177 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:55.576424 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 425308us
I20250623 01:58:55.576741 13962 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
W20250623 01:58:55.577418 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.578014 13925 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:55.572579 15477 log.cc:927] Time spent T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.192s user 0.007s sys 0.000s
I20250623 01:58:55.582626 13812 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250623 01:58:55.583211 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 440132us
I20250623 01:58:55.565128 13953 raft_consensus.cc:1273] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:55.586079 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:55.590934 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:55.581725 15380 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.064s user 0.003s sys 0.003s
I20250623 01:58:55.591584 15073 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.592216 15453 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.593698 15454 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:55.593757 15331 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:58:55.614754 15331 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:55.615145 15331 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:55.617791 15418 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:55.596246 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.596290 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:55.621678 13810 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:58:55.593022 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 272c03e6252d489cbf2fdceb36b05dc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:55.597807 15149 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.598418 15284 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.599172 14479 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
W20250623 01:58:55.581974 15491 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.068s user 0.003s sys 0.000s
I20250623 01:58:55.601411 13813 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:55.625517 15149 raft_consensus.cc:2802] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:55.625911 15149 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:55.626253 15149 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.626628 15454 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:55.628091 15433 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.604725 13958 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250623 01:58:55.605715 13807 raft_consensus.cc:2391] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:55.630385 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.631420 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.632148 13958 pending_rounds.cc:85] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 1
I20250623 01:58:55.631978 15331 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.632517 13958 pending_rounds.cc:107] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250623 01:58:55.632985 13958 raft_consensus.cc:2887] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:55.633965 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.611969 13809 raft_consensus.cc:1273] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:55.634428 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.621620 13957 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:58:55.634748 15331 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:55.635073 13879 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 2.
I20250623 01:58:55.635604 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.636487 15149 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.621623 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.637076 13958 raft_consensus.cc:2391] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:55.637771 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.638083 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.638252 13813 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:58:55.645612 13958 raft_consensus.cc:2374] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2. Re-sending same reply.
I20250623 01:58:55.636672 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.603826 13961 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:58:55.637775 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.642433 13886 raft_consensus.cc:1273] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:55.642663 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.647975 13882 raft_consensus.cc:2391] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:55.642843 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:55.646610 13806 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:58:55.645062 13956 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:55.648388 13883 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.655233 13960 raft_consensus.cc:2466] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:58:55.656010 13810 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:55.656386 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.657455 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.659247 13959 raft_consensus.cc:2374] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2. Re-sending same reply.
I20250623 01:58:55.659292 13882 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:58:55.660154 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.661042 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.657583 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.661983 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.662191 13959 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:55.662922 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.663904 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.665402 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.665990 13886 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:55.666781 13886 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:55.667760 13887 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:58:55.668582 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.669524 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:55.670053 13885 raft_consensus.cc:3053] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:55.670290 13885 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:55.671692 13888 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:55.657754 13957 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.673624 13884 raft_consensus.cc:1273] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:55.673728 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.674847 14479 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:58:55.676236 15278 raft_consensus.cc:2802] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:55.676857 15278 raft_consensus.cc:695] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [term 5 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:55.677938 15278 consensus_queue.cc:237] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.685513 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:55.685808 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.686152 15454 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:55.686630 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.686842 13884 raft_consensus.cc:2391] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:55.686046 13888 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:55.688419 15453 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:55.690012 15149 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:55.691516 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:55.692271 13811 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.695333 15358 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.696187 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:55.701305 13883 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:58:55.701649 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.703052 13882 raft_consensus.cc:2408] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250623 01:58:55.706702 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.708498 15433 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.708576 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:55.711405 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.711836 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.670768 13885 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.722406 13885 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.714532 13811 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:55.722729 13810 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.723419 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.715533 13888 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
W20250623 01:58:55.716277 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.716899 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:55.727416 14508 raft_consensus.cc:2366] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:58:55.641228 15073 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.718319 13962 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:58:55.731210 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.729813 13959 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: []
W20250623 01:58:55.736531 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.715229 13883 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:55.723991 13882 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:55.725219 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.725694 13960 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250623 01:58:55.729297 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.738749 13776 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:55.738940 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.743211 13709 catalog_manager.cc:5582] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:55.744625 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.728531 13953 raft_consensus.cc:2408] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250623 01:58:55.745206 13882 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:55.742022 13852 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:55.746187 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.750029 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.752343 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.753021 13958 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:55.721736 15316 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.757205 15316 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:55.757591 14926 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.758056 14926 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.758414 13776 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.758576 15440 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:55.759269 15446 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:55.759493 15430 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:55.759891 15440 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:55.745098 15235 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.015s
I20250623 01:58:55.760758 15235 consensus_queue.cc:1230] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer 85a58a834b6b4b0dbe197b3dad87e382 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250623 01:58:55.755894 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:55.761445 13887 raft_consensus.cc:3058] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.756353 15425 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.762253 15425 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.762655 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.756319 13879 raft_consensus.cc:1238] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:55.763823 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.756716 15284 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:55.760381 15440 raft_consensus.cc:695] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:55.761816 15073 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.728180 14479 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
W20250623 01:58:55.765033 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.754127 13810 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:58:55.771132 13853 leader_election.cc:400] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393) with higher term. Message: Invalid argument: T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:58:55.788408 13853 leader_election.cc:403] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:55.771549 13776 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:55.781276 13885 raft_consensus.cc:2408] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250623 01:58:55.784313 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.784861 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.789680 13884 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:55.790865 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:55.791446 13882 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.785429 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:55.792510 13883 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
W20250623 01:58:55.794171 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.795744 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:55.796475 13888 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.800937 13776 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.796965 13709 catalog_manager.cc:5582] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:55.808662 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.808584 13709 catalog_manager.cc:5582] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:55.807252 13886 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:55.810492 13709 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:55.812678 13852 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.813542 13776 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.811535 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52432 (request call id 166) took 1990 ms. Trace:
I20250623 01:58:55.816290 13852 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.781868 15358 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.824857 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:55.825479 13813 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:55.821915 15405 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.823272 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.827778 13709 catalog_manager.cc:5582] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:55.832724 13709 catalog_manager.cc:5582] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:55.842643 13925 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.835578 13776 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.871858 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:55.872534 13813 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:55.872530 13886 raft_consensus.cc:1273] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:58:55.835062 13709 catalog_manager.cc:5582] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:55.831421 13852 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.876114 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.801930 15446 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:55.880607 13810 pending_rounds.cc:85] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 1
I20250623 01:58:55.880924 13810 pending_rounds.cc:107] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250623 01:58:55.881170 13810 raft_consensus.cc:2887] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:55.803102 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.876713 13962 raft_consensus.cc:2374] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4. Re-sending same reply.
I20250623 01:58:55.882472 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:55.883863 13882 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:58:55.875586 13709 catalog_manager.cc:5582] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 2 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:55.886824 13709 catalog_manager.cc:5582] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 4 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:55.888693 13888 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:55.815153 13710 rpcz_store.cc:276] 0623 01:58:53.810150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:53.810380 (+ 230us) service_pool.cc:225] Handling call
0623 01:58:53.841538 (+ 31158us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b54003868d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e97bafac 00007f41e98b37ee 00007f41e7dfd624 00007f41e7dfd5b2 00007f41e7dfd4e1 00007f41e7dfd45c 00007f41e7dfc712 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:55.800896 (+1959358us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:54.599550 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:54.600093 (+ 543us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:54.600123 (+ 30us) write_op.cc:435] Acquired schema lock
0623 01:58:54.600134 (+ 11us) tablet.cc:660] Decoding operations
0623 01:58:54.601450 (+ 1316us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:54.601513 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:54.601537 (+ 24us) tablet.cc:683] Acquiring locks for 16 operations
0623 01:58:54.602671 (+ 1134us) tablet.cc:699] Row locks acquired
0623 01:58:54.602700 (+ 29us) write_op.cc:260] PREPARE: finished
0623 01:58:54.602855 (+ 155us) write_op.cc:270] Start()
0623 01:58:54.602996 (+ 141us) write_op.cc:276] Timestamp: P: 1750643934602819 usec, L: 0
0623 01:58:54.603031 (+ 35us) op_driver.cc:348] REPLICATION: starting
0623 01:58:54.603681 (+ 650us) log.cc:844] Serialized 6261 byte log entry
0623 01:58:55.321917 (+718236us) op_driver.cc:464] REPLICATION: finished
0623 01:58:55.501111 (+179194us) write_op.cc:301] APPLY: starting
0623 01:58:55.503289 (+ 2178us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:55.508886 (+ 5597us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:55.514398 (+ 5512us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:55.516200 (+ 1802us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:55.516481 (+ 281us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0623 01:58:55.516525 (+ 44us) write_op.cc:312] APPLY: finished
0623 01:58:55.516846 (+ 321us) log.cc:844] Serialized 112 byte log entry
0623 01:58:55.517016 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:55.517177 (+ 161us) write_op.cc:454] Released schema lock
0623 01:58:55.517230 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:55.565529 (+ 48299us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:55.565578 (+ 49us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12007,"reported_tablets":29,"spinlock_wait_cycles":13497088,"child_traces":[["op",{"apply.queue_time_us":164720,"apply.run_cpu_time_us":4329,"apply.run_wall_time_us":69867,"num_ops":16,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":4365,"prepare.run_wall_time_us":4365,"replication_time_us":718616,"thread_start_us":107490,"threads_started":1}]]}
I20250623 01:58:55.813089 15126 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250623 01:58:55.890179 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 54700.9us
I20250623 01:58:55.891549 14478 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.811941 13879 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:58:55.830062 15433 raft_consensus.cc:2802] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election won for term 4
W20250623 01:58:55.895148 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:55.828563 15454 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:58:55.888676 13709 catalog_manager.cc:5582] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
W20250623 01:58:55.896373 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 65064.9us
W20250623 01:58:55.816635 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 84681.9us
I20250623 01:58:55.897251 15034 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:58:55.897450 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:55.823797 15242 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:55.913106 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:55.897753 15448 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:55.915431 13852 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:55.917032 13960 raft_consensus.cc:2374] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5. Re-sending same reply.
I20250623 01:58:55.834252 15331 consensus_queue.cc:1046] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.068s
I20250623 01:58:55.901685 15416 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:55.898612 15358 raft_consensus.cc:2802] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:55.814843 15478 consensus_queue.cc:1035] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.093s
I20250623 01:58:55.926000 15416 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:55.928653 15416 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.895287 15126 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:55.912482 13879 raft_consensus.cc:604] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:55.929184 15126 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.930953 15455 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:55.931567 15455 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:55.899399 15425 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.926014 15432 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.119s
I20250623 01:58:55.901242 14926 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:55.870954 15405 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:55.914315 15446 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:55.945344 15446 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.954188 13957 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:58:55.932619 15455 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.917702 15433 raft_consensus.cc:695] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:55.897346 15483 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20250623 01:58:55.914453 15454 raft_consensus.cc:695] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:55.904623 13953 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:55.926345 15073 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:55.921309 15448 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:55.926330 15358 raft_consensus.cc:695] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:55.964448 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:55.988456 13960 raft_consensus.cc:2391] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:55.891028 15440 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.925709 13805 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:55.936143 15034 raft_consensus.cc:695] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:55.904582 15488 consensus_queue.cc:1035] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20250623 01:58:55.994048 15454 consensus_queue.cc:237] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.996672 15034 consensus_queue.cc:237] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:55.984607 15073 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:55.998309 15073 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:56.000228 15433 consensus_queue.cc:237] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:56.003736 13852 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:56.015426 13887 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:56.020773 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:56.021591 13880 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:56.018451 15397 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393) with higher term. Message: Invalid argument: T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
W20250623 01:58:56.031876 15509 log.cc:927] Time spent T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250623 01:58:55.990773 15448 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:55.974175 15446 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.004118 15242 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:56.046813 13776 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:56.019601 15444 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:56.053586 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:56.059852 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:56.066046 13960 raft_consensus.cc:1273] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250623 01:58:56.024771 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 269142us
I20250623 01:58:56.042979 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:56.048565 15242 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:58:56.071736 15504 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.259s user 0.001s sys 0.000s
I20250623 01:58:56.072285 15446 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:56.063728 15444 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:56.104055 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:56.105713 13962 raft_consensus.cc:3058] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:56.037431 14479 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:56.111083 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.112416 13812 raft_consensus.cc:1273] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:56.116371 13958 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:58:56.122457 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.123400 13812 raft_consensus.cc:2366] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:56.124262 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.126623 13808 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:56.134495 13959 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:58:56.101100 15413 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:56.100988 15488 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:56.108934 15446 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:56.135962 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:56.104888 13953 raft_consensus.cc:2374] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3. Re-sending same reply.
I20250623 01:58:56.108353 15444 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:56.163609 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:56.164216 13958 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:56.167703 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.168308 13808 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:56.169143 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.169828 13808 raft_consensus.cc:2391] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:56.174733 15413 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:56.178334 15416 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.179492 13958 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:58:56.180477 15416 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.090719 15358 consensus_queue.cc:237] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:56.189144 15446 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:56.135092 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 120110us
I20250623 01:58:56.201607 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.202320 13805 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:56.203604 13805 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250623 01:58:56.160894 15480 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:56.163059 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 626) took 1227 ms. Trace:
I20250623 01:58:56.205286 15480 raft_consensus.cc:695] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:58:56.211483 15480 consensus_queue.cc:237] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:56.212817 15405 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:56.205484 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:56.214318 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:56.215106 14508 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:56.205617 13881 rpcz_store.cc:276] 0623 01:58:54.935457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:55.290397 (+354940us) service_pool.cc:225] Handling call
0623 01:58:55.290768 (+ 371us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:55.597998 (+307230us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:55.598035 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:55.598519 (+ 484us) log.cc:844] Serialized 26 byte log entry
0623 01:58:55.728684 (+130165us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:55.731944 (+ 3260us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:55.745440 (+ 13496us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:56.151479 (+406039us) raft_consensus.cc:1683] finished
0623 01:58:56.151505 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:56.163003 (+ 11498us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":306842,"raft.queue_time_us":306956,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":154,"spinlock_wait_cycles":125824,"thread_start_us":118929,"threads_started":1,"wal-append.queue_time_us":155095}
I20250623 01:58:56.208648 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:56.215834 15488 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.218562 15488 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.210273 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.224861 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:56.228696 13962 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:58:56.170732 13808 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:56.220856 15073 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.246817 13853 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:56.247292 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:58:56.248803 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:58:56.261190 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:56.251111 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:56.266394 13961 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:56.252199 13953 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:58:56.253261 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:56.271744 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:56.272398 13959 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:58:56.253535 15488 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:56.253949 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:56.254577 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:56.255185 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:56.177191 15413 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:56.250053 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.268765 14508 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:56.277300 13956 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:56.277328 13955 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:56.281366 13812 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:56.285867 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:56.287503 13852 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:56.289534 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:56.287564 13812 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:58:56.302848 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:56.306072 15338 raft_consensus.cc:3053] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:56.306516 15338 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:56.307322 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:56.308079 13956 raft_consensus.cc:2374] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4. Re-sending same reply.
I20250623 01:58:56.289184 15316 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:56.313898 15316 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:56.314285 15316 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:56.315807 13885 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250623 01:58:56.322784 15478 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:56.323274 15478 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:56.323662 15478 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:56.296272 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:56.337378 13808 raft_consensus.cc:2366] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
W20250623 01:58:56.347311 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:56.348085 13955 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
W20250623 01:58:56.300345 15284 consensus_meta.cc:287] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.517s user 0.008s sys 0.002s
I20250623 01:58:56.353505 13812 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:58:56.356630 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.363868 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:56.370112 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 85570.3us
I20250623 01:58:56.288929 15501 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20250623 01:58:56.284863 15488 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.345911 15034 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20250623 01:58:56.369757 15454 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:56.382424 15446 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.307339 15338 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.383695 13957 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250623 01:58:56.376785 15455 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20250623 01:58:56.300526 15126 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:56.412024 15126 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:56.411968 15237 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250623 01:58:56.388118 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:56.414022 13809 raft_consensus.cc:2391] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:56.349967 13881 raft_consensus.cc:604] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:56.350010 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.400275 15381 raft_consensus.cc:991] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:58:56.366365 15284 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:56.446048 13807 raft_consensus.cc:2366] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4.
I20250623 01:58:56.449254 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:56.372916 15444 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.450891 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:56.414063 15488 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.419109 13881 raft_consensus.cc:612] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:58:56.457618 13811 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:58:56.454450 15405 raft_consensus.cc:1079] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:58:56.455915 13808 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:56.465704 15034 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:56.469108 15444 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.468957 15242 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.457396 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:56.470201 13806 raft_consensus.cc:1273] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250623 01:58:56.471288 15448 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.472275 15446 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.473840 13804 raft_consensus.cc:2366] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:56.459458 15338 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:56.474953 15073 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.415360 15454 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.476508 13806 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:56.477399 15454 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.484884 13807 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:58:56.507952 13807 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:56.482506 15448 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:56.477203 15478 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:56.471289 15316 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:56.542861 15478 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:58:56.534198 13807 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250623 01:58:56.571007 15284 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250623 01:58:56.572896 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 107830us
I20250623 01:58:56.563925 13811 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:58:56.598083 13779 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5
W20250623 01:58:56.659386 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 61404.2us
I20250623 01:58:56.662600 13779 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:56.664328 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:56.527782 15040 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
W20250623 01:58:56.676406 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 306220us
W20250623 01:58:56.676684 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 291627us
I20250623 01:58:56.558637 15106 consensus_queue.cc:1035] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
W20250623 01:58:56.686005 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 289619us
W20250623 01:58:56.685927 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 292778us
I20250623 01:58:56.684116 13779 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:56.690840 15520 log.cc:927] Time spent T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
W20250623 01:58:56.719496 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 299216us
I20250623 01:58:56.666471 15463 raft_consensus.cc:474] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Not starting pre-election -- already a leader
I20250623 01:58:56.667425 15126 raft_consensus.cc:2762] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250623 01:58:56.730450 13885 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:58:56.731711 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:56.732611 13885 raft_consensus.cc:2391] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:56.597071 15446 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.115s
I20250623 01:58:56.692461 15448 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:56.746474 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:56.748104 13886 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:56.685506 15430 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:56.671289 15488 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20250623 01:58:56.729619 15338 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:56.771144 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 87125.6us
I20250623 01:58:56.758374 15284 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 01:58:56.777107 13779 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:58:56.800565 15523 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
I20250623 01:58:56.803221 14508 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:56.776099 15126 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:58:56.810668 15440 raft_consensus.cc:2762] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250623 01:58:56.757162 15430 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:56.823109 13779 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:56.824783 13779 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:56.770766 15448 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20250623 01:58:56.768954 15338 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:56.849617 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:56.873139 13887 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250623 01:58:56.875519 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 155999us
I20250623 01:58:56.817451 15316 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:58:56.876165 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 51478.8us
I20250623 01:58:56.881984 13779 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:56.875355 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 148398us
I20250623 01:58:56.880095 13806 raft_consensus.cc:1273] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:56.886556 13779 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:56.889988 13779 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:58:56.883774 13854 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:56.869092 15126 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250623 01:58:56.897176 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 167271us
I20250623 01:58:56.864480 15284 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250623 01:58:56.892148 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:56.912719 14509 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:56.915727 13779 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:56.916303 13807 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:58:56.892438 15446 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:56.892982 15488 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:56.875763 15440 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250623 01:58:56.882714 15543 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:56.898123 15478 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:56.883025 15237 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
W20250623 01:58:57.003804 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:57.049054 15528 log.cc:927] Time spent T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20250623 01:58:57.054823 13813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1303) took 1118 ms. Trace:
I20250623 01:58:57.055207 13813 rpcz_store.cc:276] 0623 01:58:55.935977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:55.936277 (+ 300us) service_pool.cc:225] Handling call
0623 01:58:55.951299 (+ 15022us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:55.951688 (+ 389us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:55.951751 (+ 63us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:55.953794 (+ 2043us) log.cc:844] Serialized 26 byte log entry
0623 01:58:56.136635 (+182841us) spinlock_profiling.cc:243] Waited 4.81 ms on lock 0x7b18003347e0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b22bc5 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e36dd04b 00007f41e36e2b53 00007f41e36e2f01 00007f41e3a71ccf
0623 01:58:56.510458 (+373823us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:56.528194 (+ 17736us) log.cc:844] Serialized 14 byte log entry
0623 01:58:56.599465 (+ 71271us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:56.599578 (+ 113us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:57.054538 (+454960us) raft_consensus.cc:1683] finished
0623 01:58:57.054633 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:57.054753 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71964,"prepare.queue_time_us":17595,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":129365,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":17183,"spinlock_wait_cycles":11066112,"thread_start_us":373621,"threads_started":1,"wal-append.queue_time_us":916505}
I20250623 01:58:57.052858 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:57.057272 13883 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:57.057503 15555 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:57.057965 15555 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.059923 15555 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:56.907121 15425 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.094s
I20250623 01:58:57.061367 15425 consensus_queue.cc:1230] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer 85a58a834b6b4b0dbe197b3dad87e382 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250623 01:58:57.062319 15338 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:57.063843 13886 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:57.063988 13883 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:58:57.066633 15073 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:57.045045 15543 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:58:57.067657 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:57.050707 15539 log.cc:927] Time spent T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20250623 01:58:57.038098 15448 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:57.079777 13854 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:58:57.081588 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:57.082201 13812 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250623 01:58:57.038453 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 81245.8us
W20250623 01:58:57.083403 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 167709us
I20250623 01:58:57.045460 15478 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:57.089082 13779 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:58:57.089944 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 49570.7us
I20250623 01:58:57.090977 15338 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:57.091262 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:57.091552 15338 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.092097 13813 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:57.094403 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:57.095156 13812 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:57.097891 13955 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:58:57.099161 13955 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:58:57.099709 13955 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:58:57.100060 13955 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:57.100332 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:58:57.079972 15547 log.cc:927] Time spent T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20250623 01:58:57.101023 13808 raft_consensus.cc:2391] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 5: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:58:57.101572 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:57.102846 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:57.108407 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:57.118289 13812 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:57.119155 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:57.110723 13926 leader_election.cc:400] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:57.071061 15543 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:57.119545 13812 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:58:57.122979 13926 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.129209 13779 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.133127 13955 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.078299 15446 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:58:57.138891 13926 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.139767 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:57.139273 15531 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.314s user 0.000s sys 0.001s
I20250623 01:58:57.143306 13961 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:58:57.146045 13958 raft_consensus.cc:2374] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5. Re-sending same reply.
I20250623 01:58:57.112257 15478 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:57.099448 15479 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:57.145687 13955 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:58:57.150681 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 425953us
I20250623 01:58:57.153203 13779 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.156826 13813 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:57.141789 15418 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:57.153071 13807 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:57.141242 15448 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:58:57.123466 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 201219us
I20250623 01:58:57.161937 13926 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:57.174675 13926 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:57.168390 15540 log.cc:927] Time spent T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.319s user 0.000s sys 0.002s
I20250623 01:58:57.156620 15440 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:57.157182 14509 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:57.161949 15338 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:57.171507 15555 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:58:57.181975 13961 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:57.182754 13779 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.184890 13926 leader_election.cc:400] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:57.194185 13926 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:57.197530 13926 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.173676 15554 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:57.175879 15557 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20250623 01:58:57.202803 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 46762.5us
I20250623 01:58:57.182940 15556 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:57.206956 13779 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.211174 13853 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:57.203378 15558 consensus_queue.cc:1035] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20250623 01:58:57.209913 15559 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:57.214445 15126 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250623 01:58:57.214540 15542 log.cc:927] Time spent T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250623 01:58:57.215904 13776 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:57.216373 13810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1297) took 1389 ms. Trace:
I20250623 01:58:57.203492 15418 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:57.203656 15554 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:57.201112 15563 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:57.216723 13776 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:57.208725 14508 raft_consensus.cc:1238] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250623 01:58:57.216712 13810 rpcz_store.cc:276] 0623 01:58:55.826873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:55.827041 (+ 168us) service_pool.cc:225] Handling call
0623 01:58:55.880413 (+ 53372us) raft_consensus.cc:1469] Updating replica for 2 ops
0623 01:58:56.276714 (+396301us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:56.276759 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0623 01:58:56.277448 (+ 689us) log.cc:844] Serialized 52 byte log entry
0623 01:58:56.685244 (+407796us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:56.685333 (+ 89us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:56.685421 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:57.216160 (+530739us) raft_consensus.cc:1683] finished
0623 01:58:57.216181 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:57.216317 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18540,"raft.queue_time_us":22341,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":6932,"thread_start_us":665043,"threads_started":2,"wal-append.queue_time_us":734982}
I20250623 01:58:57.216131 15479 raft_consensus.cc:695] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:57.200302 15562 consensus_queue.cc:1046] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20250623 01:58:57.203498 15448 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20250623 01:58:57.210201 15565 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:57.227401 13779 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.236192 14507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36030 (request call id 15) took 1233 ms. Trace:
I20250623 01:58:57.224792 15554 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:57.226387 15126 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:57.227056 15418 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:58:57.272477 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 75022.9us
I20250623 01:58:57.221386 15237 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.044s
I20250623 01:58:57.257514 15284 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:57.260516 15448 consensus_queue.cc:1230] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20250623 01:58:57.291980 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 84413us
W20250623 01:58:57.294183 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 83119.8us
I20250623 01:58:57.266038 15548 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:57.272181 15432 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:57.284678 15126 raft_consensus.cc:695] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:57.261041 15558 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20250623 01:58:57.308869 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:57.267408 14507 rpcz_store.cc:276] 0623 01:58:56.003089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:56.014949 (+ 11860us) service_pool.cc:225] Handling call
0623 01:58:57.232936 (+1217987us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b54004cd454. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e89f509e 00007f41e209f8c7 00007f41e209f852 00007f41e209f781 00007f41e209f6fc 00007f41e209e9b2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:57.236108 (+ 3172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1496860416}
I20250623 01:58:57.322888 15418 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:57.323812 15564 log.cc:927] Time spent T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20250623 01:58:57.260852 15550 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20250623 01:58:57.294405 15572 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:57.310446 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:57.330210 13887 raft_consensus.cc:3053] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:57.330523 13887 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:57.311173 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:57.311829 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:57.312669 13885 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:57.294019 15556 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:57.313746 15548 raft_consensus.cc:695] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:57.314153 15543 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:57.317137 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 89815.1us
I20250623 01:58:57.290640 15544 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:57.318902 13811 pending_rounds.cc:85] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 1
I20250623 01:58:57.348984 13811 pending_rounds.cc:107] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250623 01:58:57.349325 13811 raft_consensus.cc:2887] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:57.340327 15126 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.265559 15565 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.342797 13779 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.317816 15479 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.337165 13887 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.362099 13954 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:57.370159 13887 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:57.366125 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:57.367470 13888 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:58:57.372411 13880 pending_rounds.cc:85] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Aborting all ops after (but not including) 1
I20250623 01:58:57.372797 13880 pending_rounds.cc:107] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250623 01:58:57.373078 13880 raft_consensus.cc:2887] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:57.378235 13779 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:57.380183 13779 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.381525 15446 consensus_queue.cc:1035] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:57.383201 15446 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:57.383781 15446 raft_consensus.cc:695] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:57.392768 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:57.397675 15552 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:57.405124 13887 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:57.409917 13926 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:57.427850 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:57.352421 15572 raft_consensus.cc:695] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:57.430482 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:57.384785 15446 consensus_queue.cc:237] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.432345 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:57.438329 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:57.439810 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:57.443603 13808 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:57.441622 13813 raft_consensus.cc:1273] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:58:57.400713 15550 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:58:57.449673 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 82792.2us
I20250623 01:58:57.452380 13810 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:57.411614 15463 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:57.368271 15554 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:57.462168 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:58:57.490227 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:57.339099 15556 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:57.356058 15544 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:57.503531 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 59552.9us
I20250623 01:58:57.447374 15278 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.052s
I20250623 01:58:57.505653 15554 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:57.508375 15242 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:57.512948 15572 consensus_queue.cc:237] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:57.412454 15577 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.088s
I20250623 01:58:57.523124 13961 raft_consensus.cc:2391] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:58:57.447825 15478 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.528769 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:57.529623 13887 raft_consensus.cc:2391] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:57.429180 15552 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.534134 15543 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:57.541247 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:57.547739 15556 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:57.620558 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:57.621421 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1218) took 1200 ms. Trace:
I20250623 01:58:57.621806 13884 rpcz_store.cc:276] 0623 01:58:56.421020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:56.421717 (+ 697us) service_pool.cc:225] Handling call
0623 01:58:57.620376 (+1198659us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:57.621357 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4656000}
W20250623 01:58:57.622840 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 169861us
I20250623 01:58:57.484370 15548 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:57.643234 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 58335.1us
I20250623 01:58:57.653048 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:50816 (request call id 2) took 1407 ms. Trace:
I20250623 01:58:57.653631 13881 rpcz_store.cc:276] 0623 01:58:56.246050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:56.247153 (+ 1103us) service_pool.cc:225] Handling call
0623 01:58:57.652762 (+1405609us) spinlock_profiling.cc:243] Waited 71.9 ms on lock 0x7b540018f954. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a99443 00007f41e89f3c4a 00007f41e209adc4 00007f41e209ad52 00007f41e209ac81 00007f41e209abfc 00007f41e2099eb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:57.652936 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165391104}
I20250623 01:58:57.622184 15550 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:57.624511 15284 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:58:57.638487 15565 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:57.703435 15585 log.cc:927] Time spent T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250623 01:58:57.706614 15478 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:57.708493 15552 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:57.711864 15544 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:57.663801 15550 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.717025 15550 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:57.766626 13808 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
W20250623 01:58:57.778499 15583 log.cc:927] Time spent T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
W20250623 01:58:57.821605 13804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 614) took 1733 ms (client timeout 1831 ms). Trace:
W20250623 01:58:57.822211 13804 rpcz_store.cc:269] 0623 01:58:56.088464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:56.090099 (+ 1635us) service_pool.cc:225] Handling call
0623 01:58:57.821335 (+1731236us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54003c64d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:57.821516 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":590984448}
W20250623 01:58:57.823925 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 177544us
W20250623 01:58:57.841954 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 272c03e6252d489cbf2fdceb36b05dc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:57.844859 14507 consensus_queue.cc:237] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:57.859778 13708 catalog_manager.cc:5582] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:57.894111 15430 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:57.905515 14479 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:57.915120 14478 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:58:57.920997 15532 log.cc:927] Time spent T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 1.108s user 0.004s sys 0.008s
I20250623 01:58:57.921509 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:57.922128 13959 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:57.922297 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 624) took 1799 ms. Trace:
I20250623 01:58:57.922678 13805 rpcz_store.cc:276] 0623 01:58:56.122686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:56.204322 (+ 81636us) service_pool.cc:225] Handling call
0623 01:58:56.204654 (+ 332us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:56.240919 (+ 36265us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:56.262111 (+ 21192us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:56.269486 (+ 7375us) log.cc:844] Serialized 26 byte log entry
0623 01:58:56.509576 (+240090us) raft_consensus.cc:1643] Marking committed up to 1
0623 01:58:56.509639 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:56.509760 (+ 121us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:57.916505 (+1406745us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:58:57.922013 (+ 5508us) raft_consensus.cc:1683] finished
0623 01:58:57.922036 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:57.922239 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":26042,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":3365,"spinlock_wait_cycles":5504256,"thread_start_us":239686,"threads_started":1,"wal-append.queue_time_us":502725}
I20250623 01:58:57.931655 13959 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:57.937805 15338 raft_consensus.cc:3053] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:57.938366 15338 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:57.939376 15338 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.940833 15338 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:58:57.959185 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 442152us
I20250623 01:58:57.959847 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:57.960467 13953 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:57.965045 15571 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:57.965662 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 342208us
I20250623 01:58:57.965596 15571 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.967595 15571 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:57.973814 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:57.976786 13888 raft_consensus.cc:1273] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:57.977433 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:57.980168 13888 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:57.993883 13882 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:57.995465 15541 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:57.996438 13852 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:57.997495 15073 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:57.998153 15073 raft_consensus.cc:695] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:57.998039 15552 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:57.999156 13879 raft_consensus.cc:1238] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:57.999085 15073 consensus_queue.cc:237] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:57.997598 13852 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:58.004259 15549 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:58.007306 13852 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:58.010051 15425 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:58.011336 15440 consensus_queue.cc:1046] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:58.012554 15590 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:58.010288 15106 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20250623 01:58:58.014955 15424 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:58.018694 13883 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:58:58.019163 13883 raft_consensus.cc:604] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:58.019404 13883 raft_consensus.cc:612] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:58:57.990329 15570 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.061s
I20250623 01:58:58.004786 15549 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.022289 15425 raft_consensus.cc:695] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:58.024011 15549 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:58.023756 15425 consensus_queue.cc:237] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:57.997531 15600 consensus_queue.cc:1035] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20250623 01:58:58.009068 15454 raft_consensus.cc:2747] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250623 01:58:58.063853 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 45973.1us
W20250623 01:58:58.062368 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 43778.6us
I20250623 01:58:58.091992 13852 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:58:57.996024 15541 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.094913 13809 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:58:58.101099 13809 raft_consensus.cc:604] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:58.015642 15424 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:58:58.062673 15483 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.380s user 0.026s sys 0.042s
I20250623 01:58:58.105785 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:58.113752 13882 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250623 01:58:58.113524 15483 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.439s user 0.026s sys 0.042s
I20250623 01:58:58.012959 15590 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.127420 13882 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:58.140830 13809 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:58:58.141819 13809 raft_consensus.cc:604] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:58:58.142252 13809 raft_consensus.cc:612] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:58:58.147924 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:58:58.121294 15567 raft_consensus.cc:3053] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:58.143569 15454 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:58.118773 15440 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20250623 01:58:58.159152 15567 raft_consensus.cc:738] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:58.160543 15454 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:58.174612 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:58.174834 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:58.175482 13811 raft_consensus.cc:2408] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250623 01:58:58.177105 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:58.181035 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:58.183880 13810 raft_consensus.cc:1238] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 2. Current term is 3. Ops: []
I20250623 01:58:58.171036 15454 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:58.181396 15592 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:58.185213 15560 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20250623 01:58:58.185678 15567 consensus_queue.cc:260] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:58.187510 15621 log.cc:927] Time spent T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20250623 01:58:58.188086 15424 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:58.182965 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:58.199548 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:58:58.208258 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:58.216955 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.221886 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:58.222594 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 65) took 2376 ms. Trace:
I20250623 01:58:58.222646 13960 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:58.223145 13708 rpcz_store.cc:276] 0623 01:58:55.832931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:55.833094 (+ 163us) service_pool.cc:225] Handling call
0623 01:58:58.209699 (+2376605us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:57.870040 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:57.870584 (+ 544us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:57.870622 (+ 38us) write_op.cc:435] Acquired schema lock
0623 01:58:57.870635 (+ 13us) tablet.cc:660] Decoding operations
0623 01:58:57.870967 (+ 332us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:57.871044 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:57.871072 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:58:57.871247 (+ 175us) tablet.cc:699] Row locks acquired
0623 01:58:57.871261 (+ 14us) write_op.cc:260] PREPARE: finished
0623 01:58:57.871396 (+ 135us) write_op.cc:270] Start()
0623 01:58:57.871500 (+ 104us) write_op.cc:276] Timestamp: P: 1750643937871350 usec, L: 0
0623 01:58:57.871517 (+ 17us) op_driver.cc:348] REPLICATION: starting
0623 01:58:57.872154 (+ 637us) log.cc:844] Serialized 529 byte log entry
0623 01:58:57.877582 (+ 5428us) op_driver.cc:464] REPLICATION: finished
0623 01:58:57.886687 (+ 9105us) write_op.cc:301] APPLY: starting
0623 01:58:57.886793 (+ 106us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:57.886887 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:57.887004 (+ 117us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:57.888820 (+ 1816us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:57.888974 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:58:57.889025 (+ 51us) write_op.cc:312] APPLY: finished
0623 01:58:57.889225 (+ 200us) log.cc:844] Serialized 22 byte log entry
0623 01:58:57.890902 (+ 1677us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:57.890987 (+ 85us) write_op.cc:454] Released schema lock
0623 01:58:57.891024 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:57.891368 (+ 344us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:57.891396 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":131076,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":8445,"apply.run_cpu_time_us":1585,"apply.run_wall_time_us":4670,"num_ops":1,"prepare.queue_time_us":4458,"prepare.run_cpu_time_us":2422,"prepare.run_wall_time_us":5284,"replication_time_us":5921,"thread_start_us":16932,"threads_started":2}]]}
I20250623 01:58:58.236126 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.236729 13961 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:58.244864 13960 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:58.210443 15418 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:58.249269 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.250077 13955 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:58.233579 15567 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:58.200142 15592 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.234293 15541 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:58.233917 15590 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:58.260535 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.246678 15489 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:58.324254 15592 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:58.328233 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 918323us
I20250623 01:58:58.321159 15489 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.329447 15544 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:58.330118 15040 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:58.329977 15544 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.331902 15544 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.332345 13953 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:58.256606 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:58:58.341111 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 91918.3us
I20250623 01:58:58.345132 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:58.345971 13886 raft_consensus.cc:2408] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250623 01:58:58.349298 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.354600 15489 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:58.355029 13953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 1303) took 1241 ms (client timeout 1504 ms). Trace:
W20250623 01:58:58.355456 13953 rpcz_store.cc:269] 0623 01:58:57.103519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:57.113219 (+ 9700us) service_pool.cc:225] Handling call
0623 01:58:57.959747 (+846528us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b5800016e98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b0323e 0000000000356d09 00007f41e897bdc0 00007f41e8a427d4 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:58:58.344515 (+384768us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6786816}
W20250623 01:58:58.357497 15626 log.cc:927] Time spent T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
W20250623 01:58:58.361552 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:58.255957 15418 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.340010 15562 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:58.375362 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 46417.6us
I20250623 01:58:58.344199 15040 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:58.395764 13926 leader_election.cc:400] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4.
I20250623 01:58:58.397253 15272 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:58.343816 15555 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:58.361562 15338 consensus_queue.cc:1046] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20250623 01:58:58.406469 15555 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.406883 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:58.407300 15338 raft_consensus.cc:3053] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:58.407648 15338 raft_consensus.cc:738] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:58:58.407969 15620 log.cc:927] Time spent T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.305s user 0.008s sys 0.000s
I20250623 01:58:58.408634 15338 consensus_queue.cc:260] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:58.401607 15418 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.402498 13925 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:58.361137 14508 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:58.409649 13881 raft_consensus.cc:1273] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:58:58.373594 15574 consensus_queue.cc:1046] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20250623 01:58:58.409910 13953 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:58:58.411664 13926 leader_election.cc:403] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:58:58.410312 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:58.440476 13883 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:58.412532 15600 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:58.415343 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:58.418385 13854 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:58.372454 15536 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:58.429584 13955 raft_consensus.cc:1273] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:58:58.409008 15555 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.433044 13885 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:58.453773 15536 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.453521 15587 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:58.456630 15536 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.457733 15574 raft_consensus.cc:3053] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:58.458246 15574 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:58.459270 15574 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.460819 15574 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:58.463399 15592 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:58:58.463949 15592 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:58.464392 15592 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:58.439891 15331 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
W20250623 01:58:58.486176 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 90554.8us
I20250623 01:58:58.429155 15040 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.394654 15562 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.486125 13888 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:58.488793 13926 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:58:58.481189 15338 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:58:58.488395 15631 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20250623 01:58:58.495636 13926 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:58:58.496415 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 48328.1us
I20250623 01:58:58.497812 13854 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:58.487957 15536 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 4
W20250623 01:58:58.511006 15629 log.cc:927] Time spent T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250623 01:58:58.490929 15331 raft_consensus.cc:695] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:58.530581 13926 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:58.494741 15555 raft_consensus.cc:2762] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250623 01:58:58.490482 15480 consensus_queue.cc:1035] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20250623 01:58:58.543712 13854 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:58:58.513947 15573 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:58.574869 15040 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.585517 13926 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:58:58.527577 15501 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:58.590076 13926 leader_election.cc:400] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:58.603216 13926 leader_election.cc:403] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250623 01:58:58.500989 15418 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
W20250623 01:58:58.590101 15629 log.cc:927] Time spent T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250623 01:58:58.584820 15575 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:58:58.609850 13960 raft_consensus.cc:1273] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:58:58.542212 15556 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:58:58.608551 15555 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:58.611994 15331 consensus_queue.cc:237] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.614446 13956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 150) took 1185 ms. Trace:
I20250623 01:58:58.602337 15573 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:58:58.540649 15272 consensus_queue.cc:1035] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20250623 01:58:58.627557 13956 rpcz_store.cc:276] 0623 01:58:57.428833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:57.428978 (+ 145us) service_pool.cc:225] Handling call
0623 01:58:57.448077 (+ 19099us) raft_consensus.cc:1469] Updating replica for 2 ops
0623 01:58:58.197190 (+749113us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:58.197237 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0623 01:58:58.201451 (+ 4214us) log.cc:844] Serialized 513 byte log entry
0623 01:58:58.335274 (+133823us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b18000b78e0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b22bc5 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e36dd04b 00007f41e36e2b53 00007f41e36e2f01 00007f41e3a71ccf
0623 01:58:58.428490 (+ 93216us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:58:58.428526 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:58.428584 (+ 58us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:58.614093 (+185509us) raft_consensus.cc:1683] finished
0623 01:58:58.614115 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:58.614367 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":858087,"raft.queue_time_us":748573,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":20042,"spinlock_wait_cycles":17933568,"thread_start_us":93042,"threads_started":1,"wal-append.queue_time_us":326410}
I20250623 01:58:58.607246 15501 raft_consensus.cc:695] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:58.645816 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:58.646878 13958 raft_consensus.cc:2391] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:58:58.649153 15558 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:58.656165 15592 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.658859 15592 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:58.635910 15455 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20250623 01:58:58.686573 15562 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.751616 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:58:58.797235 15536 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4.
I20250623 01:58:58.810065 15556 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:58.810756 15556 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:58.815088 15272 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:58.815734 15272 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.817893 15272 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.821593 15556 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:58.898602 15600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36030 (request call id 1) took 5844 ms. Trace:
I20250623 01:58:58.899358 15600 rpcz_store.cc:276] 0623 01:58:53.054608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:53.054853 (+ 245us) service_pool.cc:225] Handling call
0623 01:58:53.375633 (+320780us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b58004af6b0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a19e73 00007f41e3a7d1a7 00007f41e3a9a2fa 00007f41e3a9b35a 00007f41e3a9a50b 00007f41e3aa5c1c 00007f41e3aa5293 00007f41e3a9d5ea 00007f41e89f1b09 00007f41e208f244
0623 01:58:53.473147 (+ 97514us) log.cc:844] Serialized 487 byte log entry
0623 01:58:53.630200 (+157053us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b540056d4a0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a2fcb1 00007f41e3a18372 00007f41e3a1d781 00007f41e3a1d709 00007f41e3a1d699 00007f41e3a1d661 00007f41e3a1c82d 00007f41e9b9b124 00007f41e9b9af59 00007f41e0b40574
0623 01:58:53.701963 (+ 71763us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:58:58.898425 (+5196462us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":810,"raft.queue_time_us":33316,"raft.run_cpu_time_us":1710,"raft.run_wall_time_us":51430,"spinlock_wait_cycles":140727680,"thread_start_us":75663,"threads_started":1,"wal-append.queue_time_us":104047,"wal-append.run_cpu_time_us":3227,"wal-append.run_wall_time_us":1123547}
I20250623 01:58:58.913944 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.914633 13954 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:58:58.915534 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.916677 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.919492 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:58.921399 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:58.922335 13954 raft_consensus.cc:2408] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250623 01:58:58.934398 15536 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:58.934995 15536 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.935480 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.939110 15272 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:58.939746 15272 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.941469 15556 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:58.942034 15556 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.944234 15556 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.937156 15536 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.947363 15272 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:58.937122 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.959200 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:58.965250 13809 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:58.969630 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:58:58.971076 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 384179us
I20250623 01:58:58.994730 15536 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:58.996062 15536 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.003289 15536 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:59.006656 13959 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:58:59.008100 13959 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:59.009474 13959 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250623 01:58:59.017864 15587 consensus_meta.cc:287] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5: Time spent flushing consensus metadata: real 0.562s user 0.008s sys 0.005s
W20250623 01:58:59.020181 15338 consensus_meta.cc:287] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.524s user 0.011s sys 0.000s
I20250623 01:58:59.021212 15587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36030 (request call id 20) took 1178 ms. Trace:
I20250623 01:58:59.021771 15587 rpcz_store.cc:276] 0623 01:58:57.842857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:57.842991 (+ 134us) service_pool.cc:225] Handling call
0623 01:58:57.847877 (+ 4886us) log.cc:844] Serialized 487 byte log entry
0623 01:58:59.021118 (+1173241us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":172,"raft.run_cpu_time_us":1146,"raft.run_wall_time_us":1537,"thread_start_us":59743,"threads_started":1,"wal-append.queue_time_us":60062,"wal-append.run_cpu_time_us":3265,"wal-append.run_wall_time_us":1001573}
I20250623 01:58:59.033479 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:59.044117 13955 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
W20250623 01:58:59.076440 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 657694us
I20250623 01:58:59.081681 13954 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:58:59.095615 15575 consensus_queue.cc:1046] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:59.097404 15572 raft_consensus.cc:3053] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:58:59.097782 15572 raft_consensus.cc:738] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:59.102041 15397 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.102623 15397 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.102269 13925 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:59.104661 15338 consensus_queue.cc:1035] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:59.105368 15338 consensus_queue.cc:1230] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250623 01:58:59.098534 15572 consensus_queue.cc:260] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.113204 15272 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:59.113698 15272 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:59.114014 15272 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:59.115573 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.107223 15397 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.124555 15397 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20250623 01:58:59.129070 15454 consensus_meta.cc:287] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.928s user 0.010s sys 0.003s
I20250623 01:58:59.130201 15454 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.132282 15454 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.148200 15272 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.165530 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:59.166368 13955 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:59.166715 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:59.168550 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.169232 13805 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:59.174674 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:59.174587 15454 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:59.178133 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.180946 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.182564 13888 raft_consensus.cc:1238] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:59.183787 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.184783 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.186756 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.182852 15565 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:59.181847 15454 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.191624 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.192682 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.192739 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.189980 13879 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250623 01:58:59.195443 13879 pending_rounds.cc:85] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Aborting all ops after (but not including) 2
I20250623 01:58:59.195734 13879 pending_rounds.cc:107] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250623 01:58:59.195989 13879 raft_consensus.cc:2887] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:59.189747 15565 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.185722 15538 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:59.199546 13955 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:59.201534 15565 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.206723 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.205987 15272 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:58.688026 15501 consensus_queue.cc:237] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.214058 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:58:59.214725 14509 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:59.218868 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.219524 15572 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:59.225195 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.228103 13883 pending_rounds.cc:85] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: Aborting all ops after (but not including) 1
I20250623 01:58:59.228418 13883 pending_rounds.cc:107] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250623 01:58:59.228679 13883 raft_consensus.cc:2887] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:59.202327 15538 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.229772 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:59.233098 13888 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:59.226151 15454 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.241420 15331 consensus_queue.cc:1046] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20250623 01:58:59.252393 13954 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:58:59.254462 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 1339) took 1081 ms. Trace:
I20250623 01:58:59.254820 13809 rpcz_store.cc:276] 0623 01:58:58.173034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:58.174619 (+ 1585us) service_pool.cc:225] Handling call
0623 01:58:59.254299 (+1079680us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b54001545d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:58:59.254405 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1777880320}
W20250623 01:58:59.258872 15424 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.007s sys 0.014s
I20250623 01:58:59.258767 13854 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:59.260413 15424 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.007s sys 0.014s
I20250623 01:58:59.259374 15331 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:58:59.263536 15538 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.271618 15572 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:59.273432 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.273756 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.274070 13961 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:59.278398 13888 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:58:59.284101 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:59.272132 15572 raft_consensus.cc:695] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:59.285288 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.290026 15665 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:58:59.290469 15665 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:59.290832 15665 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:59.279145 14509 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:58:59.293783 15572 consensus_queue.cc:237] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:58.757216 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:58:59.312084 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52444 (request call id 190) took 5523 ms. Trace:
I20250623 01:58:59.312494 13709 rpcz_store.cc:276] 0623 01:58:53.788367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:53.788566 (+ 199us) service_pool.cc:225] Handling call
0623 01:58:53.848764 (+ 60198us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b54003868d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e97bafac 00007f41e98b37ee 00007f41e7dfd624 00007f41e7dfd5b2 00007f41e7dfd4e1 00007f41e7dfd45c 00007f41e7dfc712 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:58:59.312022 (+5463258us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:56.383466 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:56.384133 (+ 667us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:56.384178 (+ 45us) write_op.cc:435] Acquired schema lock
0623 01:58:56.384202 (+ 24us) tablet.cc:660] Decoding operations
0623 01:58:56.385392 (+ 1190us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:56.385497 (+ 105us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:56.385543 (+ 46us) tablet.cc:683] Acquiring locks for 10 operations
0623 01:58:56.386549 (+ 1006us) tablet.cc:699] Row locks acquired
0623 01:58:56.386570 (+ 21us) write_op.cc:260] PREPARE: finished
0623 01:58:56.386759 (+ 189us) write_op.cc:270] Start()
0623 01:58:56.386890 (+ 131us) write_op.cc:276] Timestamp: P: 1750643936386698 usec, L: 0
0623 01:58:56.386912 (+ 22us) op_driver.cc:348] REPLICATION: starting
0623 01:58:56.387638 (+ 726us) log.cc:844] Serialized 3969 byte log entry
0623 01:58:56.914470 (+526832us) op_driver.cc:464] REPLICATION: finished
0623 01:58:57.227339 (+312869us) write_op.cc:301] APPLY: starting
0623 01:58:57.228987 (+ 1648us) tablet.cc:1365] starting BulkCheckPresence
0623 01:58:57.229203 (+ 216us) tablet.cc:1368] finished BulkCheckPresence
0623 01:58:57.229342 (+ 139us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:58:57.230345 (+ 1003us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:58:57.230543 (+ 198us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0623 01:58:57.230583 (+ 40us) write_op.cc:312] APPLY: finished
0623 01:58:57.230844 (+ 261us) log.cc:844] Serialized 76 byte log entry
0623 01:58:57.231007 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:57.231128 (+ 121us) write_op.cc:454] Released schema lock
0623 01:58:57.231178 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0623 01:58:57.270769 (+ 39591us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:58:57.275947 (+ 5178us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":258141,"reported_tablets":39,"spinlock_wait_cycles":63064448,"child_traces":[["op",{"apply.queue_time_us":299342,"apply.run_cpu_time_us":2747,"apply.run_wall_time_us":32932,"num_ops":10,"prepare.queue_time_us":420713,"prepare.run_cpu_time_us":4573,"prepare.run_wall_time_us":23842,"replication_time_us":509067,"thread_start_us":327648,"threads_started":2}]]}
I20250623 01:58:59.315543 13957 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250623 01:58:59.317381 13957 pending_rounds.cc:85] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 5
I20250623 01:58:59.317638 13957 pending_rounds.cc:107] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250623 01:58:59.317870 13957 raft_consensus.cc:2887] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:58:59.353152 15600 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:59.354012 15600 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.356544 15600 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:59.371966 15669 log.cc:927] Time spent T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20250623 01:58:59.373675 15665 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.392364 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:58:59.381173 13927 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:59.393484 15665 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.394203 15418 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:58:59.394778 15418 raft_consensus.cc:695] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:59.395748 15418 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.396284 14508 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250623 01:58:59.399317 13927 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.400063 15331 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:59.394179 15665 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20250623 01:58:59.407754 15424 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:58:59.409747 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:59.414405 15034 raft_consensus.cc:3053] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:58:59.414963 15034 raft_consensus.cc:738] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:58:59.416183 15034 consensus_queue.cc:260] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.384187 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.384698 13756 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b39b05807ed544d388835cc364cb7274 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250623 01:58:59.430461 13756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 45891.1us
I20250623 01:58:59.431129 13756 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet de00fb2e24074586bfe02d33fb92e4e6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 9)
I20250623 01:58:59.431162 13958 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:58:59.433554 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:59.436966 14481 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.010s sys 0.002s
W20250623 01:58:59.438712 13756 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 0f12346da5a14097bbfa150cb8d1d907 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 7d22741cfb9b408684cb82c58241b2f5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:59.442962 14926 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:59.443523 14926 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:58:59.444365 14926 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.456089 15424 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:58:59.458671 15034 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:58:59.466162 14481 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.857s user 0.013s sys 0.002s
I20250623 01:58:59.478513 15570 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.479786 15424 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.481066 15570 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:59.480413 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 894556us
I20250623 01:58:59.481400 15424 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.503854 15570 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.506801 13962 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:58:59.512495 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.528739 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.530233 13961 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:58:59.533816 14926 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.540056 14926 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:59.544339 15648 log.cc:927] Time spent T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.549s user 0.003s sys 0.000s
I20250623 01:58:59.546733 14926 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.548187 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.562966 15424 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.563978 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.575626 14481 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:59.577589 13958 raft_consensus.cc:2953] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:59.580153 15433 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:58:59.583168 15433 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.583609 15433 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:59.587426 15424 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:59.588027 15424 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.590052 13956 raft_consensus.cc:2953] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:58:59.592609 15430 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.593082 15430 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.594918 15430 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.595327 15430 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:59.595479 15514 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:59.595775 15430 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.596025 15514 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.597885 15590 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.598371 15590 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.599452 15592 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.600973 13710 catalog_manager.cc:5582] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:58:59.604625 13879 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:58:59.605602 15515 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:59.605711 15590 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.607368 13810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 1348) took 1418 ms. Trace:
I20250623 01:58:59.610146 15430 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:59.602232 15483 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:58:59.602155 15592 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.603283 13710 catalog_manager.cc:5582] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:59.611780 15661 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.611416 15483 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.607722 13810 rpcz_store.cc:276] 0623 01:58:58.188700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:58.192230 (+ 3530us) service_pool.cc:225] Handling call
0623 01:58:58.210392 (+ 18162us) raft_consensus.cc:1469] Updating replica for 3 ops
0623 01:58:58.210902 (+ 510us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:58.210930 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0623 01:58:58.226334 (+ 15404us) log.cc:844] Serialized 78 byte log entry
0623 01:58:58.321241 (+ 94907us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:58:59.159768 (+838527us) log.cc:844] Serialized 14 byte log entry
0623 01:58:59.607066 (+447298us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:58:59.607144 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:58:59.607183 (+ 39us) raft_consensus.cc:1683] finished
0623 01:58:59.607196 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:58:59.607320 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1285295,"prepare.queue_time_us":846834,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":773,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":10142,"thread_start_us":64551,"threads_started":1,"wal-append.queue_time_us":114574}
I20250623 01:58:59.614614 13710 catalog_manager.cc:5582] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 3, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:59.617548 13710 catalog_manager.cc:5582] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 6, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:59.619575 13710 catalog_manager.cc:5582] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:59.620476 13710 catalog_manager.cc:5582] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 4, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 4 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:59.622802 13710 catalog_manager.cc:5582] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:59.623777 13710 catalog_manager.cc:5582] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 3, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:59.606158 15515 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.613898 15541 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:59.641167 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 49541.9us
I20250623 01:58:59.631788 15489 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.633574 15560 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:59.635224 15676 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:59.639158 15488 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:59.626538 15661 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.639739 15567 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.650178 15548 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.662674 15515 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:59.658895 15514 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.683220 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.655002 15433 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.648090 15541 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.688516 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:58:59.655910 15489 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.676972 15548 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.695755 15483 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:59.689208 15514 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.687701 15515 consensus_queue.cc:1046] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20250623 01:58:59.682780 15424 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.726647 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.659595 15488 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.699905 15433 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.701079 15541 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:58:59.701253 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:59.702515 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:58:59.657574 15560 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.676153 15567 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.710485 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.711315 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.711992 13882 raft_consensus.cc:1238] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:58:59.712634 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.713446 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.745173 15489 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.714388 13883 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:58:59.714301 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.715241 15691 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:59.747313 15691 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:58:59.726971 15514 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.715829 15592 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.717381 15548 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.720355 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.750552 15514 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.752086 15488 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.723487 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.725809 15661 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.727286 13954 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:58:59.658371 15676 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.755630 13779 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.756733 15676 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.756835 13779 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.757365 15676 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:58:59.757988 15676 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:58:59.737116 13960 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:58:59.758587 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.758842 15515 raft_consensus.cc:3053] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
W20250623 01:58:59.759047 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:58:59.759227 15515 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:59.733525 13958 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:58:59.743155 14481 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.758092 13779 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.766017 15687 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:58:59.768628 15418 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:59.769286 15560 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:59.736936 13834 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce4e0 after lost signal to thread 12639
W20250623 01:58:59.771400 13834 debug-util.cc:398] Leaking SignalData structure 0x7b08000d2d60 after lost signal to thread 13745
W20250623 01:58:59.772599 13834 debug-util.cc:398] Leaking SignalData structure 0x7b08002f7ec0 after lost signal to thread 13837
I20250623 01:58:59.756978 15430 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:59.763787 15592 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.774309 15592 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.776095 15592 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:59.780004 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 55346.2us
I20250623 01:58:59.780274 15691 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:59.738302 15689 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:58:59.781731 15689 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.782089 15689 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:58:59.786036 13779 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.786162 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.787292 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.788362 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.789723 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.791056 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.792855 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.794015 13854 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:59.798851 15541 raft_consensus.cc:2747] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:58:59.800940 15425 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:58:59.801380 15425 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.801754 15425 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:59.802153 13779 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.803319 13779 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.804360 13779 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.784847 15549 consensus_queue.cc:1046] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:59.805478 15665 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:58:59.806241 15676 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:59.806535 15541 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:58:59.806867 15676 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:58:59.806815 15665 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:59.807216 15541 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:58:59.807822 15549 consensus_queue.cc:1230] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250623 01:58:59.807847 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.808588 15541 raft_consensus.cc:3053] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:58:59.808558 13958 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:58:59.809047 15541 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:58:59.812433 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.764056 15040 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:59.813053 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.827507 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:58:59.815443 15515 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:58:59.831445 15515 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:59.833474 15662 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:58:59.833897 15662 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.834250 15662 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:58:59.820149 15040 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.837826 15040 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:58:59.839068 15541 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.840550 15541 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:58:59.820303 13834 debug-util.cc:398] Leaking SignalData structure 0x7b0800329140 after lost signal to thread 13912
I20250623 01:58:59.813978 15567 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.847110 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.847833 13884 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:58:59.848841 13925 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:58:59.844735 13834 debug-util.cc:398] Leaking SignalData structure 0x7b0800375420 after lost signal to thread 13913
I20250623 01:58:59.850744 15515 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:58:59.842346 15549 consensus_queue.cc:1162] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c243715fe12445019c33bbeafbaea635" responder_term: 3 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20250623 01:58:59.851357 15515 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:59.851725 13834 debug-util.cc:398] Leaking SignalData structure 0x7b0800391660 after lost signal to thread 13986
I20250623 01:58:59.853303 15515 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:58:59.853415 13834 debug-util.cc:398] Leaking SignalData structure 0x7b08003b1100 after lost signal to thread 14540
I20250623 01:58:59.853804 15515 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.850850 15272 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:58:59.859238 15272 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:59.859653 15272 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:58:59.854195 13834 debug-util.cc:398] Leaking SignalData structure 0x7b08003a4fc0 after lost signal to thread 15413
I20250623 01:58:59.862814 15425 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:58:59.863528 13834 debug-util.cc:398] Leaking SignalData structure 0x7b08002b6a00 after lost signal to thread 15444
I20250623 01:58:59.855698 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.856150 15592 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:58:59.857704 13779 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:58:59.857898 14508 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20250623 01:58:59.870203 13834 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0c40 after lost signal to thread 15593
I20250623 01:58:59.827464 15488 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
W20250623 01:58:59.872398 13834 debug-util.cc:398] Leaking SignalData structure 0x7b0800331d00 after lost signal to thread 15594
I20250623 01:58:59.879407 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.879945 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:58:59.880033 13808 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:59.880259 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.880687 13810 raft_consensus.cc:2366] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:58:59.854264 15515 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.873464 15590 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:58:59.881541 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.873667 15686 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.882488 15515 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.882668 15686 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.883678 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.884249 13807 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 1.
I20250623 01:58:59.884588 15686 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.873862 15567 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.885485 15567 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.882323 13811 raft_consensus.cc:2408] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250623 01:58:59.886281 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.887269 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.887315 15567 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.882910 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.887915 13813 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:58:59.880834 13812 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:59.888669 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.889003 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.892244 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.894912 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.896908 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.897193 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.896450 13806 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:58:59.899223 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.905550 15425 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.871383 15592 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.874924 15661 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.907534 15662 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.907987 15592 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.908262 15661 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.910207 15661 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.875095 15489 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:58:59.910791 15489 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.912719 15489 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.913441 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.913369 15034 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:58:59.914875 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.915864 15483 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:58:59.905442 13805 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:58:59.916364 15483 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:58:59.922971 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:58:59.923509 13961 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:58:59.923321 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:58:59.928427 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:58:59.938396 15662 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:58:59.939323 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:00.059520 13852 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.1:34781 timed out after 1.760s (SENT) [suppressed 17 similar messages]
I20250623 01:59:00.059885 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.061789 13961 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:58:59.918303 15488 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:00.085801 15488 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.086266 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.087301 13807 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:00.088054 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:00.088348 15488 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:00.068837 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 142628us
I20250623 01:59:00.072048 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:00.089193 15689 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.090871 13813 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 7: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:00.091120 15689 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:00.093250 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.094305 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.074028 15272 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.074877 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.100833 13806 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:00.102146 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:58:59.874732 15549 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.086s
I20250623 01:59:00.103597 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.105132 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.105962 13806 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:00.076692 15483 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:00.096302 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:00.088883 13807 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:00.095247 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.107048 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.111450 13960 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:00.112866 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:00.115557 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:00.123347 13813 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:59:00.123741 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.125869 13807 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:00.123725 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.133776 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.135161 13807 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:00.135589 13888 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:00.136770 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.143482 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:00.144333 13807 raft_consensus.cc:2408] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250623 01:59:00.156522 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:00.102281 15661 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:00.159166 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:00.160840 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:00.164585 13956 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:00.167259 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.167906 13807 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:00.168605 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.168687 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.167884 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.169171 13807 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:00.169641 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.169669 13954 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:00.170306 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.169803 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.173768 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.175967 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.178490 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.179867 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.181617 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.185034 13807 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:00.189916 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.196060 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.204237 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.281090 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:00.247012 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 61157.3us
I20250623 01:59:00.248440 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:00.247736 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 62594.2us
I20250623 01:59:00.248865 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:00.277586 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 89163.4us
W20250623 01:59:00.281006 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:00.282490 13960 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:00.286815 13957 raft_consensus.cc:1238] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20250623 01:59:00.245851 15706 log.cc:927] Time spent T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20250623 01:59:00.186074 13854 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:00.246008 13807 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:00.181917 15661 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:00.292116 15700 log.cc:927] Time spent T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
I20250623 01:59:00.299562 13813 raft_consensus.cc:1273] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:59:00.300815 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.357144 13807 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:00.360055 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.300037 15661 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:00.363916 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:00.299247 15272 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:00.304481 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:00.300154 15424 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:00.369400 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.373896 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:00.374636 13888 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:00.296726 15040 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
W20250623 01:59:00.380640 15648 log.cc:927] Time spent T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20250623 01:59:00.396762 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 81749.6us
I20250623 01:59:00.397010 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:00.397452 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 211445us
I20250623 01:59:00.398404 13888 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:00.423142 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.384127 13805 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:00.387625 15424 raft_consensus.cc:695] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:00.445518 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.460287 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:00.465094 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.469826 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:00.473615 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.474359 13954 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250623 01:59:00.474804 13779 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:00.475934 13779 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:00.468123 15703 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
W20250623 01:59:00.477285 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:00.464043 15418 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:00.478178 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:00.478772 13779 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:00.472954 15702 log.cc:927] Time spent T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20250623 01:59:00.480243 13779 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.509235 13926 leader_election.cc:400] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:59:00.510185 15040 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:59:00.510033 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:00.517189 15040 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:00.517683 15040 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:00.510769 15541 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:00.509624 13926 leader_election.cc:403] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250623 01:59:00.511005 13779 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.523610 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:00.523954 13779 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:00.572332 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 48499.9us
W20250623 01:59:00.572433 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 48901.7us
I20250623 01:59:00.572965 13779 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.530120 13778 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.574250 13926 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.508699 15567 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:00.576181 13926 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:59:00.571818 15669 log.cc:927] Time spent T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20250623 01:59:00.577955 13926 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.471467 15424 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.531869 14506 raft_consensus.cc:1238] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:00.479596 15661 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:59:00.574185 15515 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:00.523108 15034 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5.
I20250623 01:59:00.529302 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:00.528807 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:00.578855 15573 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:00.508502 15272 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:00.587276 13779 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.583401 15567 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:00.478582 15549 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:59:00.576642 15562 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:00.640216 15562 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:00.640771 15562 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.479688 15483 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:00.525482 15430 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:59:00.637236 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 50002.5us
I20250623 01:59:00.575642 15331 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:59:00.590798 15691 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:00.645792 13779 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.638674 15515 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:59:00.714785 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 312179us
I20250623 01:59:00.632051 14509 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
W20250623 01:59:00.744527 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 98848.8us
I20250623 01:59:00.770987 13779 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.509490 15418 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.772562 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250623 01:59:00.778172 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 61859.9us
I20250623 01:59:00.783370 13779 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:00.785997 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 208256us
I20250623 01:59:00.786818 13926 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:00.787189 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 159854us
W20250623 01:59:00.787846 13778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 257821us
I20250623 01:59:00.643152 15034 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:00.795113 13926 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.796756 13926 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:00.636405 15573 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:00.645004 15483 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:00.803781 13779 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.717374 15331 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:00.809079 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:00.782886 15689 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:00.814046 15715 consensus_queue.cc:1046] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:00.814426 13779 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.815299 15693 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:59:00.815860 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 167693us
W20250623 01:59:00.817394 15705 log.cc:927] Time spent T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.453s user 0.001s sys 0.005s
I20250623 01:59:00.838521 15040 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.839150 15562 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:00.840626 15040 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:00.805470 15483 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:00.805541 15686 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:00.645509 15272 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.799787 15515 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:00.826357 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:00.827581 13778 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:00.770010 15430 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:00.806898 15573 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:00.844054 13852 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:00.811546 15331 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:00.828516 15034 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250623 01:59:00.954804 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 140503us
W20250623 01:59:00.870780 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 44509.6us
W20250623 01:59:00.956748 13778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 129272us
W20250623 01:59:00.957473 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 97277.6us
I20250623 01:59:00.958786 13953 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:00.961076 15541 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:00.962005 15541 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:00.962453 15541 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:00.967983 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:00.971154 15728 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:00.971588 15728 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:00.971932 15728 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:00.972610 15691 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:00.950212 13879 raft_consensus.cc:1238] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:00.865788 13852 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:00.983289 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 117603us
I20250623 01:59:00.993750 13778 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:00.955580 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:00.999228 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 43731.2us
I20250623 01:59:01.000186 13779 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.001585 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:01.088686 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 81293.6us
I20250623 01:59:00.953378 15430 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:00.953898 15478 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:01.101951 15478 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:01.102578 15478 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:01.104933 15478 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:01.086133 15515 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.019570 15665 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:01.087307 15661 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:01.109054 15696 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:01.021437 15489 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:00.950614 15562 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.182s
I20250623 01:59:01.112010 15331 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.112816 13956 raft_consensus.cc:1238] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:59:00.847858 15730 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:00.953578 15040 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:01.136641 15541 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:01.138576 15689 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:01.156335 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 154837us
I20250623 01:59:01.157403 13779 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.159313 13779 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.150759 15696 raft_consensus.cc:695] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:01.160449 15661 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.155937 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:01.172050 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 375336us
I20250623 01:59:01.237749 15696 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.239522 13926 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.240267 15040 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.240694 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.240773 15040 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:01.241988 15726 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:01.243192 15040 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:01.243837 15726 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:01.243247 13926 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.246625 13926 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:01.246970 15661 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:01.247327 15562 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.247988 15562 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250623 01:59:01.248154 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 88944.5us
I20250623 01:59:01.249156 15034 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:01.156185 15489 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.174844 15418 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.172005 15430 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.201177 13882 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:01.164829 15689 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.252269 15689 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.208725 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.200018 15425 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20250623 01:59:01.155102 15730 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.255944 15730 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.256240 15726 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:01.257550 15425 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:01.258132 15425 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.260090 15425 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:01.278632 15699 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.913s user 0.005s sys 0.015s
I20250623 01:59:01.279965 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1417) took 1464 ms. Trace:
I20250623 01:59:01.279989 15726 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:01.281777 15726 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:01.280885 15704 log.cc:927] Time spent T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.915s user 0.007s sys 0.027s
I20250623 01:59:01.283968 15726 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:01.284492 15726 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.290335 15726 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.292151 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:01.198554 13888 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:01.280362 13958 rpcz_store.cc:276] 0623 01:58:59.815242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:59.815920 (+ 678us) service_pool.cc:225] Handling call
0623 01:58:59.828403 (+ 12483us) raft_consensus.cc:1469] Updating replica for 2 ops
0623 01:58:59.828847 (+ 444us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:59.828898 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0623 01:58:59.829639 (+ 741us) log.cc:844] Serialized 52 byte log entry
0623 01:59:00.249324 (+419685us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:59:00.249378 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:00.249448 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:01.278064 (+1028616us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:01.279691 (+ 1627us) raft_consensus.cc:1683] finished
0623 01:59:01.279711 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:01.279911 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6437,"raft.queue_time_us":14976,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":3625,"spinlock_wait_cycles":63851264,"thread_start_us":378615,"threads_started":1,"wal-append.queue_time_us":514773}
I20250623 01:59:01.292981 14509 raft_consensus.cc:2391] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:01.293025 15726 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.293644 13888 raft_consensus.cc:604] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.298200 15562 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.300205 15562 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.301409 15331 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.302899 15562 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.295240 15483 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.306701 15573 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.307873 13954 raft_consensus.cc:1238] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:01.308446 15483 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.308864 15573 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.318428 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:01.249296 13779 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:01.323074 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 73872.8us
I20250623 01:59:01.328672 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.336091 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:01.338501 15701 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 1.032s user 0.005s sys 0.021s
I20250623 01:59:01.340570 15430 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:01.341125 15514 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.342114 15425 raft_consensus.cc:991] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:01.342624 15425 raft_consensus.cc:1079] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:01.343315 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 649) took 1464 ms. Trace:
I20250623 01:59:01.330194 15483 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:01.300282 15728 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.343699 13779 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.347306 15728 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.348098 15483 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:59:01.348836 15698 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 1.044s user 0.005s sys 0.011s
I20250623 01:59:01.343880 13809 rpcz_store.cc:276] 0623 01:58:59.878803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:59.881226 (+ 2423us) service_pool.cc:225] Handling call
0623 01:58:59.881516 (+ 290us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:58:59.881798 (+ 282us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:58:59.881824 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:58:59.882221 (+ 397us) log.cc:844] Serialized 26 byte log entry
0623 01:59:00.105919 (+223698us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:59:00.166559 (+ 60640us) log.cc:844] Serialized 14 byte log entry
0623 01:59:00.270821 (+104262us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:00.270890 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:01.331977 (+1061087us) spinlock_profiling.cc:243] Waited 87.3 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:01.342942 (+ 10965us) raft_consensus.cc:1683] finished
0623 01:59:01.343067 (+ 125us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:01.343239 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105448,"prepare.queue_time_us":164743,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":3406,"raft.queue_time_us":113626,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":11247,"spinlock_wait_cycles":200703872,"thread_start_us":169439,"threads_started":1,"wal-append.queue_time_us":363242}
I20250623 01:59:01.385298 15592 consensus_queue.cc:1046] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:59:01.386355 15592 raft_consensus.cc:3053] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:59:01.386703 15592 raft_consensus.cc:738] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:01.387058 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.387419 15592 consensus_queue.cc:260] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.388264 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.389462 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:01.398952 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 414415us
I20250623 01:59:01.380223 15483 consensus_queue.cc:1046] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.014s
I20250623 01:59:01.380476 15514 consensus_queue.cc:1046] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20250623 01:59:01.409708 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:01.395778 15592 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:01.397455 15567 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:01.423893 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.424515 13954 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:01.428180 13954 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:01.428894 13954 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:01.429350 13954 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:01.436229 13888 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3828994fa152441a95c5034d002516db"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:01.436843 13958 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:59:01.436870 13854 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:01.438017 15662 raft_consensus.cc:3053] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:01.438398 15662 raft_consensus.cc:738] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:01.436839 13888 raft_consensus.cc:604] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:01.439182 15662 consensus_queue.cc:260] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:01.440501 15662 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:01.441663 13888 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:01.442282 13888 raft_consensus.cc:604] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.444119 15455 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:01.444584 15455 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:01.444751 13888 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:01.445112 15455 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:01.445413 13888 raft_consensus.cc:604] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.447942 13888 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:01.448488 13888 raft_consensus.cc:604] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:01.453878 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:59:01.454620 15735 consensus_queue.cc:1046] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250623 01:59:01.455523 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:01.456354 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:01.418154 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:01.461808 15723 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:01.462504 15723 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.463371 15691 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.464025 15691 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.464586 15515 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.469710 15691 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.469390 15515 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.450291 15567 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.472456 15567 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.421327 15430 consensus_queue.cc:1046] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20250623 01:59:01.421803 15489 raft_consensus.cc:3053] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:01.473402 15489 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:01.474241 15489 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.474723 15515 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.421659 15689 consensus_queue.cc:1046] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20250623 01:59:01.475718 15689 consensus_queue.cc:1230] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250623 01:59:01.476280 15723 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:01.480741 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 47058.4us
I20250623 01:59:01.482508 13852 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.500831 15272 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.523218 13954 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.506480 15743 consensus_queue.cc:1046] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250623 01:59:01.524449 13956 raft_consensus.cc:1238] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:59:01.526357 13954 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.517820 15478 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:01.539592 13808 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20250623 01:59:01.540083 13808 pending_rounds.cc:85] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 4
I20250623 01:59:01.540369 13808 pending_rounds.cc:107] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20250623 01:59:01.540647 13808 raft_consensus.cc:2887] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250623 01:59:01.541992 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 59549.2us
I20250623 01:59:01.545286 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.542850 13852 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.545951 13808 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:01.553160 13808 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:01.554817 15661 raft_consensus.cc:3053] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:01.554986 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.555259 15661 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:01.555568 15665 raft_consensus.cc:3053] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:01.555987 15665 raft_consensus.cc:738] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:01.556027 15661 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.557719 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:01.556502 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:01.558581 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:01.559301 13809 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:01.560849 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:01.563134 13811 raft_consensus.cc:2391] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:01.563900 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:01.564590 13808 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:01.566566 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:01.567374 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.568452 13811 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:01.569489 13808 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:01.570730 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.571784 13807 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:01.536027 15455 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.583221 13807 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:01.596076 13811 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:01.597399 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.598007 13808 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:01.602046 13779 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.572821 15661 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:01.556850 15665 consensus_queue.cc:260] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.615960 13808 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:59:01.573402 15541 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.622210 13926 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.622267 13852 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.628165 13926 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.629626 13926 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.632229 15541 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.635053 15541 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.591192 15712 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:01.609216 15689 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:01.614835 15430 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250623 01:59:01.646286 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 52755.2us
W20250623 01:59:01.645177 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 54196.2us
I20250623 01:59:01.645639 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.650183 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.650712 13888 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:01.656520 13852 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.657100 15552 consensus_queue.cc:1046] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:59:01.658058 13852 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.658285 15552 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:01.658869 15552 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:01.659188 15552 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:01.659593 15552 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:01.660082 13852 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.661814 15749 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.662413 15743 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:01.665364 13926 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.637451 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.677397 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.643218 15272 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:01.661473 15748 raft_consensus.cc:3053] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:59:01.678582 13926 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.683128 13926 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.685619 13926 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.687839 15747 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:01.688809 13852 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.689546 15747 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:01.689915 13852 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.690459 15747 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:01.691689 15425 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:01.692334 15425 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:01.694154 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:01.695166 15731 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:01.691908 15747 raft_consensus.cc:3053] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:01.696754 15747 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:01.697515 15747 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.698699 15747 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:01.701822 15730 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:01.703198 13888 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:01.706043 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.708904 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.709647 13888 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:01.675997 15726 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:01.681774 15562 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:01.710346 15726 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.647991 15689 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.713620 15686 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.682304 15272 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:01.684911 15748 raft_consensus.cc:738] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:01.714299 15686 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.715667 15748 consensus_queue.cc:260] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.718919 15686 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.713328 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.710427 15562 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:01.720800 15562 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:01.722304 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.723661 13879 raft_consensus.cc:2391] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:01.642637 15573 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250623 01:59:01.649912 15727 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:01.673924 15665 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:01.710966 15726 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:01.714354 15272 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:01.727442 13804 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:01.730844 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.731069 15489 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:01.714293 15455 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.731464 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.740649 15742 raft_consensus.cc:3053] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
W20250623 01:59:01.749608 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 64086.9us
I20250623 01:59:01.750442 15040 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:59:01.741039 15742 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:01.752063 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:01.752219 13956 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 2. Ops: []
I20250623 01:59:01.753522 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.754446 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.755354 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.768015 13853 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.756150 13953 raft_consensus.cc:1238] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250623 01:59:01.757295 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.758175 13776 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.759752 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.775133 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.776320 13776 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.783919 13776 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.781019 15455 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:01.781567 15483 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:59:01.800349 15040 raft_consensus.cc:695] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:01.802386 13958 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:01.791914 15567 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.785833 15727 raft_consensus.cc:695] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:01.794420 15686 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.814728 15040 consensus_queue.cc:237] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.795807 15515 consensus_queue.cc:1046] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20250623 01:59:01.810650 15730 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:01.826925 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 136021us
I20250623 01:59:01.827955 13776 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.829416 13776 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.813895 15488 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:01.842190 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.850586 13776 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.852571 15478 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.811874 15455 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.029s
I20250623 01:59:01.808094 15272 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.842101 15686 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:01.858927 13776 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:01.859361 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 45158.9us
I20250623 01:59:01.866299 13852 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:01.856310 15691 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:01.867007 13960 raft_consensus.cc:2366] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:01.867897 13776 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:01.856230 15723 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:01.841327 15712 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20250623 01:59:01.878650 13811 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:01.880820 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.884060 13954 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:01.809398 15742 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.887332 13813 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:01.854379 15488 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.869335 15731 raft_consensus.cc:695] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:01.868866 15691 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:01.929987 15691 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:01.925124 13804 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:59:01.932263 15430 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:01.932500 15272 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:01.932752 15430 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.883329 15483 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:59:01.889323 15686 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:01.934028 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.934610 13813 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:01.917167 15552 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.874826 15478 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20250623 01:59:01.937364 15552 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.913895 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:01.840729 15730 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.886300 15740 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:01.894868 15592 consensus_queue.cc:1046] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20250623 01:59:01.941731 15740 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:01.942157 15740 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:01.943378 15756 log.cc:927] Time spent T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.169s user 0.003s sys 0.000s
I20250623 01:59:01.943846 15430 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.885675 15743 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250623 01:59:01.945884 15488 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:01.946559 15541 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:01.946961 15541 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:01.949999 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.866948 15727 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.868672 15752 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
W20250623 01:59:01.951529 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:01.915716 15757 log.cc:927] Time spent T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20250623 01:59:01.953495 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:01.953286 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.954219 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:01.966712 13888 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:01.956313 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:01.967113 15562 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.967790 13776 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:01.969472 15488 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250623 01:59:01.969830 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 71458.9us
I20250623 01:59:01.971803 15237 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:01.972287 15719 raft_consensus.cc:991] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:01.972757 15719 raft_consensus.cc:1079] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:01.969871 15488 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:01.974601 15488 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:01.925165 15726 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:01.983891 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 272c03e6252d489cbf2fdceb36b05dc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:01.972321 15237 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:01.986336 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.988801 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:01.956821 13813 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:01.947744 15541 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:01.987088 13962 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:01.989601 13958 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:01.994843 13926 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:01.996673 13779 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:02.000638 13779 leader_election.cc:400] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:02.000921 13779 leader_election.cc:403] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:01.949898 15731 consensus_queue.cc:237] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:02.006417 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.007020 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:01.969210 15562 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:02.010058 13813 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.013547 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:02.018031 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.018649 13887 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:01.962935 15747 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20250623 01:59:02.001195 15272 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:59:02.022356 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.004930 15686 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:59:02.027508 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.028091 13883 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:02.028402 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.028891 13885 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:02.031103 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.034871 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.015761 15728 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:02.023909 15723 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:02.038373 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.038462 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.009958 15567 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:02.043747 13813 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:02.044960 13926 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:02.046305 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.054544 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.055629 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.056561 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.057596 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.062551 15723 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.064934 13852 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:02.065171 15237 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.065812 15723 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.068063 15773 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:02.069649 15665 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:02.070617 15728 raft_consensus.cc:695] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:02.026180 15686 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:02.074903 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.075624 13885 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:02.076563 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.077695 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.078418 14478 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:02.074443 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:02.012200 15740 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.081956 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:02.083029 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.087678 13883 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:02.084244 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.087050 15570 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250623 01:59:02.085301 13887 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:59:02.071561 15728 consensus_queue.cc:237] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.097391 13925 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:02.037388 15272 raft_consensus.cc:695] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:02.109747 15748 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:02.110507 15272 consensus_queue.cc:237] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.111733 15767 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:02.112278 15767 raft_consensus.cc:695] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:02.096309 15541 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:02.100975 13808 raft_consensus.cc:1238] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:02.114017 13779 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.096349 13884 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:02.102198 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:02.096926 15237 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:02.048113 15726 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:02.085006 15686 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:02.128891 13883 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:02.131800 13925 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:02.133116 13882 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:02.149636 13813 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3828994fa152441a95c5034d002516db"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:02.150224 13813 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:02.150588 13813 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.151366 15767 consensus_queue.cc:237] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.122856 15237 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:02.156559 13813 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.136092 15483 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:02.166082 13813 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.170964 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:02.171525 13958 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.160665 15237 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:02.160295 15726 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250623 01:59:02.213557 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 52269.1us
I20250623 01:59:02.214805 15752 raft_consensus.cc:991] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:02.215165 15712 raft_consensus.cc:991] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:02.165797 15761 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:02.217650 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.246129 13880 raft_consensus.cc:3053] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:02.246507 13880 raft_consensus.cc:738] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:02.207363 13804 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250623 01:59:02.224236 13958 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:02.236797 15726 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:02.247666 13852 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:02.253419 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.247365 13880 consensus_queue.cc:260] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.262243 15726 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:02.253050 15749 raft_consensus.cc:1079] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:02.245594 15761 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:02.254966 15662 raft_consensus.cc:1079] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:02.262610 15744 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:02.255820 15691 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.255424 15488 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.278921 15761 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:02.256055 15696 consensus_queue.cc:1046] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20250623 01:59:02.299263 13812 raft_consensus.cc:1238] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:02.287076 15740 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:02.310468 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 176167us
I20250623 01:59:02.314417 15237 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:02.315649 13925 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.318095 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:59:02.313778 15783 log.cc:927] Time spent T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.064s user 0.001s sys 0.001s
I20250623 01:59:02.319787 15237 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.325591 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.328850 13883 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:02.343513 13883 raft_consensus.cc:2466] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:02.345008 14478 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
W20250623 01:59:02.346097 15779 log.cc:927] Time spent T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.097s user 0.000s sys 0.003s
I20250623 01:59:02.346190 15237 raft_consensus.cc:2802] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:02.346812 15237 raft_consensus.cc:695] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:02.347687 15237 consensus_queue.cc:237] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:02.359812 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 120281us
I20250623 01:59:02.382633 15665 raft_consensus.cc:3053] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:02.383283 15665 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:59:02.364107 15778 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.121s user 0.001s sys 0.001s
I20250623 01:59:02.331633 15744 consensus_queue.cc:1046] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20250623 01:59:02.407593 13886 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
W20250623 01:59:02.427109 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 111941us
I20250623 01:59:02.444701 15744 raft_consensus.cc:3053] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:02.445196 15744 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:02.445952 15744 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.447248 15744 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.446936 13925 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.438766 13961 raft_consensus.cc:1238] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:02.435288 13958 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:02.472498 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:02.436929 13953 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:02.478132 13925 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.426870 15728 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:02.478539 13953 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:02.478977 13953 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.384289 15665 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.443034 15689 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.408821 15691 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.409373 15488 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.476181 15764 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:02.498870 15732 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:02.499429 15732 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:02.501451 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 268830us
I20250623 01:59:02.505808 15483 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:02.506325 15483 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.491137 13953 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.485180 15686 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.508361 15483 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:02.496310 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 277880us
I20250623 01:59:02.510824 15726 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:02.511409 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.512720 15726 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:02.515727 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.516896 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.517602 13884 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.517999 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:02.518540 13962 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.518951 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:02.520792 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:02.520870 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:02.524120 13958 raft_consensus.cc:2391] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:02.527530 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.518253 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:02.508960 15483 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:02.557549 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:02.542061 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 65758.3us
I20250623 01:59:02.558552 13888 raft_consensus.cc:2391] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 2: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:02.558728 13813 raft_consensus.cc:1273] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:59:02.559176 13925 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.541947 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.562081 13958 raft_consensus.cc:1238] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:02.562461 13925 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:02.536242 13779 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:02.563707 15764 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:02.567365 13953 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:02.568090 13962 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:02.509052 15686 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.585784 13884 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:02.590408 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.560948 15726 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:02.590998 13887 raft_consensus.cc:2366] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4.
W20250623 01:59:02.591428 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.592799 13925 leader_election.cc:400] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4.
I20250623 01:59:02.593107 13925 leader_election.cc:403] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250623 01:59:02.594720 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.590162 15725 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:59:02.595288 13888 raft_consensus.cc:3053] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:02.595688 13888 raft_consensus.cc:738] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:02.597172 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:02.600875 13812 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:02.603089 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:02.596407 13888 consensus_queue.cc:260] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.605773 13888 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.609539 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:02.576840 15732 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.611299 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:02.607019 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 70915.5us
I20250623 01:59:02.612370 13776 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:02.516417 15761 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:02.595820 15573 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4.
I20250623 01:59:02.611065 13809 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:02.620360 14926 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:02.595707 15725 raft_consensus.cc:695] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:02.571153 15483 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:02.617455 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.615171 15763 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:02.635880 15725 consensus_queue.cc:237] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.636632 15763 raft_consensus.cc:695] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:02.637538 15763 consensus_queue.cc:237] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.632290 13852 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:02.634166 15752 consensus_queue.cc:1046] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:02.648200 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:02.650674 15759 raft_consensus.cc:3053] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:02.651093 15759 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:02.616039 15726 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:02.591439 15767 consensus_queue.cc:1046] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20250623 01:59:02.636368 15761 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:02.629902 15567 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:02.641228 15719 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:02.670703 15719 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:02.671092 15719 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:02.670861 15726 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.651895 15759 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.672942 15726 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:02.642210 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:02.673539 15759 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:02.643577 15768 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:02.645100 15483 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:02.679901 15768 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.634775 13805 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:02.656718 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.682017 15768 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:02.682227 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.682576 13882 raft_consensus.cc:1273] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:02.682935 13883 raft_consensus.cc:2366] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7.
I20250623 01:59:02.683568 15567 raft_consensus.cc:695] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:02.683959 15719 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:02.688618 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.690652 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.670250 15767 raft_consensus.cc:3053] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:02.690832 13925 leader_election.cc:400] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7.
I20250623 01:59:02.691272 13925 leader_election.cc:403] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250623 01:59:02.692185 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.693981 13776 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.696182 13882 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:02.696539 13882 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:02.697185 13882 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.618937 13888 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
W20250623 01:59:02.711217 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 93882.7us
I20250623 01:59:02.712186 13779 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.712713 13925 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:02.718165 13779 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.722676 13779 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.709782 15768 raft_consensus.cc:3058] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:02.708987 15483 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:02.719292 15689 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:02.685106 15567 consensus_queue.cc:237] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.704869 15767 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:02.715910 15726 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:02.737728 13813 raft_consensus.cc:1238] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:02.739418 13957 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:02.740407 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:02.740962 13807 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.724187 15723 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:59:02.743366 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:02.744046 13811 raft_consensus.cc:1273] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:02.744925 15767 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.746155 15767 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:02.750607 15764 consensus_queue.cc:1046] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:02.750691 15788 consensus_queue.cc:1035] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:59:02.751775 15764 raft_consensus.cc:3053] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:02.752138 15764 raft_consensus.cc:738] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:02.753427 15726 raft_consensus.cc:695] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:02.754838 15726 consensus_queue.cc:237] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.716881 15784 consensus_queue.cc:1035] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
W20250623 01:59:02.759812 15787 consensus_queue.cc:1162] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "85a58a834b6b4b0dbe197b3dad87e382" responder_term: 5 status { last_received { term: 5 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 5 index: 3 } }
I20250623 01:59:02.715859 15752 consensus_queue.cc:1046] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20250623 01:59:02.729483 15777 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:02.736294 15661 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250623 01:59:02.764369 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.766034 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:02.766770 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:02.767287 13809 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:02.767315 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:02.773644 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.776798 13807 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:02.770146 13805 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 1. Current term is 3. Ops: []
W20250623 01:59:02.782933 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.789789 13926 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.802686 13807 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:59:02.807508 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:02.810801 15719 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.812811 13926 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:02.814453 13812 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:02.814983 13812 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:02.817095 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.820519 13809 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:02.821434 13852 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:02.798096 15787 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:02.752923 15764 consensus_queue.cc:260] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.838456 13810 raft_consensus.cc:1238] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:02.838573 15768 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7.
I20250623 01:59:02.788271 15727 consensus_queue.cc:1035] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20250623 01:59:02.843824 13884 raft_consensus.cc:1273] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:59:02.828442 15665 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.821725 15784 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20250623 01:59:02.831867 15689 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 01:59:02.832147 15742 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:02.824883 15725 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:02.834151 13882 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:02.836210 15719 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:02.861792 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:02.867997 13812 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.854681 15430 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:02.871790 13958 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250623 01:59:02.872479 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:02.873941 13882 raft_consensus.cc:3053] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:02.874214 13882 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:02.874747 13882 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.862795 15719 raft_consensus.cc:695] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:02.862999 15728 consensus_queue.cc:1035] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20250623 01:59:02.884063 13812 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:02.880544 15708 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20250623 01:59:02.878103 15430 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:02.903924 15430 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:02.906286 15719 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:02.918980 15764 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:02.958190 13879 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:02.960963 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:02.968552 13882 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:02.970762 13879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 1320) took 1021 ms. Trace:
I20250623 01:59:02.983093 13879 rpcz_store.cc:276] 0623 01:59:01.949332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:01.953059 (+ 3727us) service_pool.cc:225] Handling call
0623 01:59:02.970497 (+1017438us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b540029bdd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:02.970677 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2279735680}
I20250623 01:59:02.977072 13709 catalog_manager.cc:5582] T de00fb2e24074586bfe02d33fb92e4e6 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 3 to 6, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New cstate: current_term: 4 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250623 01:59:02.986914 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 102550us
W20250623 01:59:02.991292 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:02.997254 13709 catalog_manager.cc:5582] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:03.017015 13957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 1473) took 1078 ms. Trace:
I20250623 01:59:03.017757 13957 rpcz_store.cc:276] 0623 01:59:01.938455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:01.942312 (+ 3857us) service_pool.cc:225] Handling call
0623 01:59:01.945194 (+ 2882us) spinlock_profiling.cc:243] Waited 916 us on lock 0x7b5800016e98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b0323e 0000000000356d09 00007f41e897bdc0 00007f41e8a427d4 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:59:01.945356 (+ 162us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:02.744047 (+798691us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:03.016334 (+272287us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54000bccd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:03.016766 (+ 432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1827889280}
I20250623 01:59:03.021209 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:03.023623 13778 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:03.032589 13811 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:03.034323 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:03.035079 13958 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:03.036019 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:03.047423 13884 raft_consensus.cc:1273] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250623 01:59:03.051342 15430 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:03.052976 15433 consensus_queue.cc:1035] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:03.055998 13880 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:03.011878 13882 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:03.115448 13958 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
W20250623 01:59:03.116588 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 47219.2us
W20250623 01:59:03.124955 15662 consensus_queue.cc:1162] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "85a58a834b6b4b0dbe197b3dad87e382" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250623 01:59:03.130015 13957 raft_consensus.cc:1238] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:03.133113 15730 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:03.133749 13808 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:59:03.138376 13776 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:03.139712 15515 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:03.140349 15515 raft_consensus.cc:695] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:03.141741 15515 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.136303 13709 catalog_manager.cc:5582] T b39b05807ed544d388835cc364cb7274 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2), VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3), VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 4 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:03.156042 13880 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:03.171259 13709 catalog_manager.cc:5582] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 1 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 5 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:03.188031 13887 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:03.189291 15331 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:03.190150 15331 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:03.191040 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 67671.6us
W20250623 01:59:03.202786 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 45176.5us
I20250623 01:59:03.204521 13887 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:59:03.205973 13883 raft_consensus.cc:1273] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250623 01:59:03.207878 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:03.211622 13888 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:03.213678 15730 consensus_queue.cc:1046] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250623 01:59:03.255229 15725 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:03.262682 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 173) took 6476 ms. Trace:
I20250623 01:59:03.282476 13812 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:03.293666 15740 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:03.294484 15727 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:59:03.296041 15727 consensus_queue.cc:1046] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:03.294486 15740 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.297703 15764 raft_consensus.cc:3053] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:03.298194 15764 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:03.299284 15740 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:03.299396 15764 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.302371 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:03.303021 13957 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:03.303184 15573 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:03.304143 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:03.303977 15573 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.305809 15665 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250623 01:59:03.296352 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 127826us
I20250623 01:59:03.306234 15665 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:03.306648 15665 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:03.314841 13879 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:59:03.243041 15331 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:03.319994 13880 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:03.263270 13710 rpcz_store.cc:276] 0623 01:58:56.786571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:56.795300 (+ 8729us) service_pool.cc:225] Handling call
0623 01:59:03.262427 (+6467127us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:58:59.827248 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:58:59.841871 (+ 14623us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:58:59.841904 (+ 33us) write_op.cc:435] Acquired schema lock
0623 01:58:59.841925 (+ 21us) tablet.cc:660] Decoding operations
0623 01:58:59.857250 (+ 15325us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:58:59.857302 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0623 01:58:59.857359 (+ 57us) tablet.cc:683] Acquiring locks for 8 operations
0623 01:58:59.857963 (+ 604us) tablet.cc:699] Row locks acquired
0623 01:58:59.857981 (+ 18us) write_op.cc:260] PREPARE: finished
0623 01:58:59.858111 (+ 130us) write_op.cc:270] Start()
0623 01:58:59.858237 (+ 126us) write_op.cc:276] Timestamp: P: 1750643939858072 usec, L: 0
0623 01:58:59.858258 (+ 21us) op_driver.cc:348] REPLICATION: starting
0623 01:58:59.858886 (+ 628us) log.cc:844] Serialized 3209 byte log entry
0623 01:59:01.284687 (+1425801us) spinlock_profiling.cc:243] Waited 9.98 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:01.503203 (+218516us) op_driver.cc:464] REPLICATION: finished
0623 01:59:01.546039 (+ 42836us) write_op.cc:301] APPLY: starting
0623 01:59:01.546114 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:01.546292 (+ 178us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:01.546317 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:01.547376 (+ 1059us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:01.547668 (+ 292us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0623 01:59:01.547735 (+ 67us) write_op.cc:312] APPLY: finished
0623 01:59:01.548071 (+ 336us) log.cc:844] Serialized 64 byte log entry
0623 01:59:01.578014 (+ 29943us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:01.593373 (+ 15359us) write_op.cc:454] Released schema lock
0623 01:59:01.611587 (+ 18214us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:01.654555 (+ 42968us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:01.664365 (+ 9810us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":42581,"apply.run_cpu_time_us":3131,"apply.run_wall_time_us":97156,"num_ops":8,"prepare.queue_time_us":166350,"prepare.run_cpu_time_us":15332,"prepare.run_wall_time_us":244861,"raft.queue_time_us":152974,"raft.run_cpu_time_us":15126,"raft.run_wall_time_us":43484,"replication_time_us":1644764,"spinlock_wait_cycles":22942592,"thread_start_us":393697,"threads_started":4,"wal-append.queue_time_us":306000,"wal-append.run_cpu_time_us":65467,"wal-append.run_wall_time_us":2309831}]]}
I20250623 01:59:03.332382 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:03.333424 13886 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:03.333802 13885 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:03.338668 13812 raft_consensus.cc:1273] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:59:03.340130 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:03.340911 13809 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:03.347697 13810 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:03.350767 15728 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:59:03.380911 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:03.381708 13809 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:03.371798 15725 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20250623 01:59:03.372339 15730 consensus_queue.cc:1046] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.022s
I20250623 01:59:03.405752 15573 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:03.416735 13957 raft_consensus.cc:1273] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:03.421432 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:03.435688 15731 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20250623 01:59:03.445858 13882 raft_consensus.cc:2391] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:03.429098 15730 consensus_queue.cc:1230] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250623 01:59:03.461915 15787 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20250623 01:59:03.464447 15726 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250623 01:59:03.462543 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:03.460979 15724 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
W20250623 01:59:03.531213 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 103456us
I20250623 01:59:03.538781 15761 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:03.553475 13888 raft_consensus.cc:1273] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:03.529980 15665 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:03.561436 15826 log.cc:927] Time spent T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250623 01:59:03.562958 15834 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:03.563627 15834 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.557220 15761 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:03.565011 15761 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:03.569840 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:03.552640 15772 consensus_queue.cc:1046] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20250623 01:59:03.579475 15764 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:03.581502 13879 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 8: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:03.588344 15567 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:03.594965 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:03.597949 13810 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:03.588358 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:03.602298 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:03.607074 13809 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250623 01:59:03.597834 15567 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20250623 01:59:03.612048 13888 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
W20250623 01:59:03.616600 15821 log.cc:927] Time spent T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.324s user 0.005s sys 0.000s
I20250623 01:59:03.618144 13806 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:03.618662 15665 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:03.619968 15727 consensus_queue.cc:1046] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:59:03.621472 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:03.622308 13960 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:03.627130 13958 raft_consensus.cc:3053] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:03.628955 13958 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:03.629725 15767 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:03.629846 13958 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:03.630208 13756 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 0f12346da5a14097bbfa150cb8d1d907 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 7d22741cfb9b408684cb82c58241b2f5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:03.630383 15767 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.631260 13958 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:03.632375 15767 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:03.628188 13776 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:03.642179 13926 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:03.691294 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:59:03.691330 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 55806.4us
W20250623 01:59:03.695086 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 54934.1us
I20250623 01:59:03.698485 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:03.608798 15834 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:03.696627 15761 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.709189 15835 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:03.691167 15728 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:03.746073 13809 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:03.753436 13954 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:03.721666 15835 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:03.728371 15728 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:03.820650 15784 raft_consensus.cc:3053] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:03.811913 15835 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:03.832723 15728 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:03.832578 15784 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:03.848351 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 130227us
I20250623 01:59:03.860920 13961 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:03.871995 13958 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
W20250623 01:59:03.872684 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 273848us
I20250623 01:59:03.872406 13958 pending_rounds.cc:85] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 3
I20250623 01:59:03.876833 13958 pending_rounds.cc:107] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250623 01:59:03.877097 13958 raft_consensus.cc:2887] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:03.877750 13958 pending_rounds.cc:107] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20250623 01:59:03.878015 13958 raft_consensus.cc:2887] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250623 01:59:03.894711 15040 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20250623 01:59:03.897694 15661 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:03.903529 15761 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:03.912420 15759 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:03.914610 15784 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.916050 15784 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:03.917495 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:03.918218 13881 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:03.926375 13885 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:03.934274 15849 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:03.936403 13810 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
W20250623 01:59:03.943511 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 59712.6us
I20250623 01:59:03.954425 13960 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:03.957435 15728 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.959499 15728 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:03.960489 13806 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:03.961700 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:03.963387 13811 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 8: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:03.964795 13852 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:03.966727 15662 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:03.968225 15424 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:03.968314 15662 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.968760 15424 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:03.969702 15424 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.970636 15662 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:03.973433 15557 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:03.974032 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:03.976837 13806 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:03.956634 15835 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:03.993764 13852 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:03.998116 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 48446.4us
I20250623 01:59:04.008661 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:03.997881 15759 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:04.034332 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.035053 13962 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:04.028077 15759 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:04.042507 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:04.051998 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:04.052791 13962 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:04.053005 15851 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:04.053625 15851 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.056241 15851 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.040891 15835 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.073200 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:04.073441 13812 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:04.048774 15759 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.079107 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:04.080624 13879 raft_consensus.cc:2391] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:04.052280 15849 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20250623 01:59:04.087275 13926 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:04.092156 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.091275 15728 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:04.094569 13925 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:04.099947 15772 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:04.100334 15772 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:04.100715 15772 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:04.104295 15728 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:04.105540 15728 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.114490 13962 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:04.118026 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:04.151616 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:04.154950 13779 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:04.174815 13885 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:04.159744 13925 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:04.193506 13776 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
W20250623 01:59:04.217147 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:04.181586 15849 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250623 01:59:04.233151 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 46595.3us
I20250623 01:59:04.250437 15850 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:04.252794 15840 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:04.232170 15849 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:04.278237 15850 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:04.284188 15849 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.291021 15759 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.310510 13881 raft_consensus.cc:2366] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6.
I20250623 01:59:04.311751 15759 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.284754 15772 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.314616 15772 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:04.320860 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:04.305150 15850 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.327946 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:04.328548 13813 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.335672 13888 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:04.343369 13813 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:04.348978 13852 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:04.353673 14926 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:04.354251 14926 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:04.355123 14926 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:04.368358 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 48467.1us
I20250623 01:59:04.371728 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:04.372287 13925 leader_election.cc:400] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6.
I20250623 01:59:04.372591 15040 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:04.373020 13925 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:04.377744 15764 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:04.378273 15764 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:04.379176 15764 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.375902 15040 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.385856 15040 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:04.398600 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:04.399345 13810 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:04.402390 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:04.403492 13810 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250623 01:59:04.429124 15849 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.430495 15712 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:04.431051 15712 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.432969 15712 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.434070 15773 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:04.434669 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:04.435395 13885 raft_consensus.cc:2391] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:04.432991 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:04.439340 13882 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:04.451431 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:04.451633 13925 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:04.452716 15040 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:04.453161 15040 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:04.453583 15040 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:04.451792 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.462298 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 66) took 4892 ms. Trace:
I20250623 01:59:04.466643 13803 tablet_service.cc:1515] Processing DeleteTablet for tablet de00fb2e24074586bfe02d33fb92e4e6 with delete_type TABLET_DATA_TOMBSTONED (TS 85a58a834b6b4b0dbe197b3dad87e382 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48562
I20250623 01:59:04.454437 13954 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:04.443135 15849 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.480625 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:04.462735 13709 rpcz_store.cc:276] 0623 01:58:59.570313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:59.570593 (+ 280us) service_pool.cc:225] Handling call
0623 01:59:04.462221 (+4891628us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:03.436669 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:03.437532 (+ 863us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:03.437589 (+ 57us) write_op.cc:435] Acquired schema lock
0623 01:59:03.437611 (+ 22us) tablet.cc:660] Decoding operations
0623 01:59:03.443094 (+ 5483us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:03.443209 (+ 115us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:03.443255 (+ 46us) tablet.cc:683] Acquiring locks for 4 operations
0623 01:59:03.443991 (+ 736us) tablet.cc:699] Row locks acquired
0623 01:59:03.444016 (+ 25us) write_op.cc:260] PREPARE: finished
0623 01:59:03.444192 (+ 176us) write_op.cc:270] Start()
0623 01:59:03.444332 (+ 140us) write_op.cc:276] Timestamp: P: 1750643943444127 usec, L: 0
0623 01:59:03.444364 (+ 32us) op_driver.cc:348] REPLICATION: starting
0623 01:59:03.445076 (+ 712us) log.cc:844] Serialized 1965 byte log entry
0623 01:59:04.138949 (+693873us) op_driver.cc:464] REPLICATION: finished
0623 01:59:04.335615 (+196666us) write_op.cc:301] APPLY: starting
0623 01:59:04.335729 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:04.335930 (+ 201us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:04.337600 (+ 1670us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:04.338277 (+ 677us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:04.338568 (+ 291us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0623 01:59:04.338644 (+ 76us) write_op.cc:312] APPLY: finished
0623 01:59:04.338958 (+ 314us) log.cc:844] Serialized 40 byte log entry
0623 01:59:04.347717 (+ 8759us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:04.350956 (+ 3239us) write_op.cc:454] Released schema lock
0623 01:59:04.359356 (+ 8400us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:04.397534 (+ 38178us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:04.397563 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1314464,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":196320,"apply.run_cpu_time_us":3018,"apply.run_wall_time_us":61896,"num_ops":4,"prepare.queue_time_us":239166,"prepare.run_cpu_time_us":14785,"prepare.run_wall_time_us":167672,"raft.queue_time_us":209089,"raft.run_cpu_time_us":66140,"raft.run_wall_time_us":155291,"replication_time_us":694378,"thread_start_us":570768,"threads_started":4,"wal-append.queue_time_us":362040}]]}
I20250623 01:59:04.486478 13954 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250623 01:59:04.490034 15744 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:04.490402 15744 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:04.490711 15744 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.493659 15712 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:04.484038 13708 catalog_manager.cc:5582] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:04.518255 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:04.523388 15040 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.523887 13957 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:04.525102 15689 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:04.525288 13779 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:04.525710 15689 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.527784 15689 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.528282 15689 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:04.528739 15689 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:04.529665 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.530292 13954 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:04.530025 15689 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.531816 13779 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:04.533712 15567 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:04.534072 15567 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:04.534381 15567 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.506273 13708 catalog_manager.cc:5582] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 5 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:04.553858 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:04.557124 15040 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:04.564010 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:04.564620 13882 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:04.572739 13882 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:04.559638 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:04.561808 13708 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 5, leader changed from 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 5 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:04.592669 13925 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:04.593580 15040 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:04.596132 15040 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:04.501289 15744 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.597401 13708 catalog_manager.cc:5582] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 4 to 5, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:04.625221 15040 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.620196 15430 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:04.629657 15430 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.619072 15724 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:04.633620 15724 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.635248 13810 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:04.635425 15724 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.616788 13708 catalog_manager.cc:5582] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index 3 to 5, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted, no longer has a pending config: VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2), VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3), VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 4 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:04.643549 15772 consensus_queue.cc:1046] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:04.644898 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.654508 13962 raft_consensus.cc:2408] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250623 01:59:04.656138 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:04.661151 15874 tablet_replica.cc:331] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:04.662429 15874 raft_consensus.cc:2241] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:04.662809 15567 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.663798 13708 catalog_manager.cc:5582] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 5, leader changed from 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:04.644371 15772 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:04.669265 15772 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:04.670106 15772 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.671240 15772 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.673048 15430 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.673205 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:04.673678 15874 raft_consensus.cc:2270] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:04.675822 13881 raft_consensus.cc:2408] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250623 01:59:04.675876 15665 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:04.676556 15665 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.678200 15567 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.678462 15665 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.680030 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.680648 13961 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:04.680941 13779 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:04.682668 13854 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:04.685497 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.687687 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:04.688357 13813 raft_consensus.cc:2366] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:04.689283 15665 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:04.689728 15665 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:04.690110 15665 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:04.691713 15430 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:04.694113 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:04.695572 13779 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:04.696202 13852 leader_election.cc:400] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:04.698110 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:04.698635 13961 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.702055 15665 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.695434 15850 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:04.703151 13882 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.703568 15850 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.682286 13708 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 4 to 5, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:04.705487 15850 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.705637 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:04.706594 13806 raft_consensus.cc:2366] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:04.711685 15874 ts_tablet_manager.cc:1905] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:04.711934 13961 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:04.712425 15689 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:04.710256 13852 leader_election.cc:400] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:04.712419 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:04.713377 13852 leader_election.cc:403] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250623 01:59:04.713732 13881 raft_consensus.cc:2366] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:04.715269 13776 leader_election.cc:400] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:04.715548 13776 leader_election.cc:403] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:04.716274 15689 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.719661 15744 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.721189 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:04.721469 13779 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:04.721854 13961 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:04.721138 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:04.705651 13708 catalog_manager.cc:5582] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } }
I20250623 01:59:04.723843 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.725520 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:04.726163 13957 raft_consensus.cc:2366] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:04.722870 13882 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:04.728021 13854 leader_election.cc:400] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:04.728070 15567 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:04.728487 15665 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.734539 15749 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:04.737118 15567 raft_consensus.cc:695] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:04.747251 13961 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:04.748204 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:04.751183 15557 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:04.751672 15557 raft_consensus.cc:695] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:04.751582 15567 consensus_queue.cc:237] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.748986 13708 catalog_manager.cc:5582] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 6, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:04.755609 13708 catalog_manager.cc:5582] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:04.752708 15557 consensus_queue.cc:237] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.760526 13881 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:04.759231 13708 catalog_manager.cc:5582] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:04.764524 13961 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:04.768395 13708 catalog_manager.cc:5582] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:04.779145 13708 catalog_manager.cc:5582] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:04.780615 15489 consensus_queue.cc:1046] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20250623 01:59:04.781446 15489 raft_consensus.cc:3053] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:04.781795 15489 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:04.781865 13708 catalog_manager.cc:5582] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 3, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:04.782459 15489 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.783449 15489 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:04.783169 13708 catalog_manager.cc:5582] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:04.784864 13708 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
W20250623 01:59:04.786022 15849 consensus_queue.cc:1162] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c243715fe12445019c33bbeafbaea635" responder_term: 7 status { last_received { term: 7 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20250623 01:59:04.811450 15877 log.cc:927] Time spent T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.078s user 0.003s sys 0.000s
I20250623 01:59:04.814002 15689 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:04.846330 15773 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:04.846952 15773 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.850893 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:04.854570 13960 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:04.849860 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:04.857199 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:04.858001 15665 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:04.849761 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:04.858575 15665 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:04.859594 13881 raft_consensus.cc:604] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:04.859727 15665 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:04.860009 13881 raft_consensus.cc:612] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:04.849277 15773 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.871505 13810 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:04.879102 13882 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:04.890663 13960 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250623 01:59:04.898834 13960 pending_rounds.cc:85] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 3
I20250623 01:59:04.899173 13960 pending_rounds.cc:107] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250623 01:59:04.899556 13960 raft_consensus.cc:2887] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:04.867835 13806 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:04.878391 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:04.897758 15782 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:04.902070 15740 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:04.905537 15784 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250623 01:59:04.918081 13881 raft_consensus.cc:604] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:04.921916 15665 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.926071 15665 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:04.927047 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:04.927629 13961 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:04.934504 13885 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3828994fa152441a95c5034d002516db"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:04.935101 13885 raft_consensus.cc:604] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:04.935434 13885 raft_consensus.cc:612] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:04.936786 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:04.939894 13810 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:04.942600 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:04.943336 13881 raft_consensus.cc:604] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:04.943749 13881 raft_consensus.cc:612] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:04.943807 13961 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:04.941931 15719 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:04.947440 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:04.954988 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f44146f651124d8fba4cc1825f480603"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:04.955605 13881 raft_consensus.cc:604] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:04.959367 15557 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:04.959926 15557 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:04.960793 15557 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:04.962512 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:59:04.964202 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:04.965190 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:05.028267 15874 ts_tablet_manager.cc:1918] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20250623 01:59:05.041479 15874 log.cc:1199] T de00fb2e24074586bfe02d33fb92e4e6 P 85a58a834b6b4b0dbe197b3dad87e382: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root/wals/de00fb2e24074586bfe02d33fb92e4e6
I20250623 01:59:05.053460 13695 catalog_manager.cc:4928] TS 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): tablet de00fb2e24074586bfe02d33fb92e4e6 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:05.065276 13882 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:05.067672 15851 consensus_queue.cc:1046] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:05.068420 15851 consensus_queue.cc:1230] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250623 01:59:05.068989 15689 raft_consensus.cc:3053] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:05.069371 15689 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:05.070170 15689 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.072863 13961 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:05.083420 15689 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:05.112277 13882 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:05.113925 15784 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:05.165405 13806 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20250623 01:59:05.165776 13806 pending_rounds.cc:85] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 4
I20250623 01:59:05.166069 13806 pending_rounds.cc:107] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20250623 01:59:05.166272 13806 raft_consensus.cc:2887] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:05.165805 13709 catalog_manager.cc:5582] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 5 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 7 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:05.181769 13961 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:05.171074 15784 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20250623 01:59:05.184093 13803 tablet_service.cc:1515] Processing DeleteTablet for tablet b39b05807ed544d388835cc364cb7274 with delete_type TABLET_DATA_TOMBSTONED (TS 85a58a834b6b4b0dbe197b3dad87e382 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48562
I20250623 01:59:05.188019 15874 tablet_replica.cc:331] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:05.188938 15874 raft_consensus.cc:2241] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:05.189642 15874 raft_consensus.cc:2270] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:05.195015 15874 ts_tablet_manager.cc:1905] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:05.196708 13809 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:05.185375 15689 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250623 01:59:05.200781 15665 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250623 01:59:05.207415 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52444 (request call id 191) took 5545 ms. Trace:
I20250623 01:59:05.217913 13708 rpcz_store.cc:276] 0623 01:58:59.662057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:58:59.662202 (+ 145us) service_pool.cc:225] Handling call
0623 01:59:05.207359 (+5545157us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:04.989442 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:04.997780 (+ 8338us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:04.997814 (+ 34us) write_op.cc:435] Acquired schema lock
0623 01:59:04.997831 (+ 17us) tablet.cc:660] Decoding operations
0623 01:59:04.999730 (+ 1899us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:04.999804 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:04.999835 (+ 31us) tablet.cc:683] Acquiring locks for 16 operations
0623 01:59:05.000970 (+ 1135us) tablet.cc:699] Row locks acquired
0623 01:59:05.000984 (+ 14us) write_op.cc:260] PREPARE: finished
0623 01:59:05.001115 (+ 131us) write_op.cc:270] Start()
0623 01:59:05.001222 (+ 107us) write_op.cc:276] Timestamp: P: 1750643945001072 usec, L: 0
0623 01:59:05.001243 (+ 21us) op_driver.cc:348] REPLICATION: starting
0623 01:59:05.001936 (+ 693us) log.cc:844] Serialized 6307 byte log entry
0623 01:59:05.036973 (+ 35037us) op_driver.cc:464] REPLICATION: finished
0623 01:59:05.108267 (+ 71294us) write_op.cc:301] APPLY: starting
0623 01:59:05.108356 (+ 89us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:05.108681 (+ 325us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:05.108798 (+ 117us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:05.141473 (+ 32675us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:05.141769 (+ 296us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0623 01:59:05.141838 (+ 69us) write_op.cc:312] APPLY: finished
0623 01:59:05.144921 (+ 3083us) log.cc:844] Serialized 112 byte log entry
0623 01:59:05.145076 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:05.145225 (+ 149us) write_op.cc:454] Released schema lock
0623 01:59:05.145278 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:05.145653 (+ 375us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:05.145685 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2064,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":71053,"apply.run_cpu_time_us":4164,"apply.run_wall_time_us":37370,"num_ops":16,"prepare.queue_time_us":176580,"prepare.run_cpu_time_us":4761,"prepare.run_wall_time_us":35727,"replication_time_us":35573,"thread_start_us":151169,"threads_started":2}]]}
I20250623 01:59:05.228920 13710 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 6, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.235196 13710 catalog_manager.cc:5582] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:05.246438 13710 catalog_manager.cc:5582] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.248646 15874 ts_tablet_manager.cc:1918] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250623 01:59:05.252826 13710 catalog_manager.cc:5582] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.255445 13710 catalog_manager.cc:5582] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.259197 15752 raft_consensus.cc:991] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:05.259716 15749 raft_consensus.cc:1079] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:05.255759 15874 log.cc:1199] T b39b05807ed544d388835cc364cb7274 P 85a58a834b6b4b0dbe197b3dad87e382: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root/wals/b39b05807ed544d388835cc364cb7274
I20250623 01:59:05.261220 13809 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:05.260919 13710 catalog_manager.cc:5582] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 4, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 4 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:05.262210 13809 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:05.265522 13809 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:05.266549 13960 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:05.270442 13695 catalog_manager.cc:4928] TS 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): tablet b39b05807ed544d388835cc364cb7274 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:05.270925 15749 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:05.266266 13710 catalog_manager.cc:5582] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:05.274919 13710 catalog_manager.cc:5582] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.279098 13710 catalog_manager.cc:5582] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.286302 13710 catalog_manager.cc:5582] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:05.290659 13710 catalog_manager.cc:5582] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.292909 13710 catalog_manager.cc:5582] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.298977 13710 catalog_manager.cc:5582] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.300436 13710 catalog_manager.cc:5582] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.304281 13710 catalog_manager.cc:5582] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:05.306079 13710 catalog_manager.cc:5582] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 2 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.308020 13710 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:05.310834 13882 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:05.313781 15430 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:05.303540 13809 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.317286 13806 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:05.318873 13809 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:05.318869 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:05.319577 13881 raft_consensus.cc:3053] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:05.319828 13881 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:05.321758 13881 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.325227 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:05.327486 15752 consensus_queue.cc:1046] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.008s
I20250623 01:59:05.342474 13881 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:05.369943 13881 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:05.373492 13776 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:05.374733 15851 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:05.377758 15851 raft_consensus.cc:695] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:05.378865 15851 consensus_queue.cc:237] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.427604 15665 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:05.428221 15665 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.430603 15665 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:05.436759 13806 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250623 01:59:05.437172 13806 pending_rounds.cc:85] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 3
I20250623 01:59:05.437458 13806 pending_rounds.cc:107] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250623 01:59:05.437690 13806 raft_consensus.cc:2887] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250623 01:59:05.438454 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 46400.6us
I20250623 01:59:05.436969 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:05.440109 13810 raft_consensus.cc:2391] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:05.440001 15740 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20250623 01:59:05.448179 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:05.450011 13953 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:05.453449 13854 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:05.455037 15719 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:59:05.455394 15719 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:05.455749 15719 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:05.465250 13953 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:05.478057 15773 raft_consensus.cc:991] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:05.478510 15773 raft_consensus.cc:1079] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:05.483731 13806 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f44146f651124d8fba4cc1825f480603"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:05.484308 13806 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:05.485271 13806 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:05.488605 15749 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:05.489154 15719 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.491122 15719 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:05.492699 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:05.494943 13810 raft_consensus.cc:2391] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:05.496618 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:05.524513 13806 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.530532 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:05.531145 13882 raft_consensus.cc:3053] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:05.531455 13882 raft_consensus.cc:738] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:05.532205 13882 consensus_queue.cc:260] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.543280 13882 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:05.526368 13954 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:05.571722 13854 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:05.572928 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:05.573194 15749 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:05.573683 13806 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:05.573817 15749 raft_consensus.cc:695] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:05.575053 15749 consensus_queue.cc:237] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.577980 13709 catalog_manager.cc:5582] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 4 to 6, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.580114 13709 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 5 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:05.584928 13709 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 4 to 6, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.574066 13954 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:05.596215 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52432 (request call id 167) took 5230 ms. Trace:
I20250623 01:59:05.570477 13882 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:05.596892 13803 tablet_service.cc:1515] Processing DeleteTablet for tablet de00fb2e24074586bfe02d33fb92e4e6 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:48562
I20250623 01:59:05.596860 13709 catalog_manager.cc:5582] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 4 to 6, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:05.606978 13695 catalog_manager.cc:4928] TS 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): tablet de00fb2e24074586bfe02d33fb92e4e6 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:05.608055 13776 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:05.609097 15851 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:05.609627 15851 raft_consensus.cc:695] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:05.609288 13709 catalog_manager.cc:5582] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:05.610402 15851 consensus_queue.cc:237] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:05.613377 13711 rpcz_store.cc:276] 0623 01:59:00.366334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:00.366572 (+ 238us) service_pool.cc:225] Handling call
0623 01:59:05.596038 (+5229466us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41877,"reported_tablets":35}
I20250623 01:59:05.627365 13954 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:05.704849 13711 catalog_manager.cc:5582] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 7, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:05.707690 13711 catalog_manager.cc:5582] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 3, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:05.811028 13709 catalog_manager.cc:5582] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 2 to 3, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:05.814779 13709 catalog_manager.cc:5582] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 6 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 7 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:05.910401 13953 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:05.930223 13953 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:05.932785 15567 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:05.989992 13882 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:05.993072 15567 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:06.015275 13954 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:06.029341 15719 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:59:06.082778 13885 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:59:06.084415 15851 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:59:06.118507 13959 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250623 01:59:06.120298 15567 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:59:06.150172 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250623 01:59:06.220829 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 52339.2us
I20250623 01:59:06.233031 13810 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:06.236632 15749 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:59:06.360024 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:06.360632 13881 raft_consensus.cc:604] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:06.376464 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:06.377036 13881 raft_consensus.cc:604] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:06.379824 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:06.380368 13881 raft_consensus.cc:604] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:06.382797 13885 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:06.383340 13885 raft_consensus.cc:604] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:06.385742 13885 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:06.386262 13885 raft_consensus.cc:604] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:06.388116 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:59:06.393683 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:06.394361 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:06.423501 15908 raft_consensus.cc:991] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:06.423964 15908 raft_consensus.cc:1079] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:06.425767 13809 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:06.426420 13809 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:06.426983 13809 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:06.446602 13809 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.540729 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:06.541404 13885 raft_consensus.cc:3053] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:06.541712 13885 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:06.542416 13885 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.543546 13885 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:06.555423 13885 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:06.556653 13776 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:06.557416 15430 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:06.557940 15430 raft_consensus.cc:695] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:06.558789 15430 consensus_queue.cc:237] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.576607 15719 raft_consensus.cc:991] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:06.577174 15908 raft_consensus.cc:1079] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:06.578980 13961 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:06.579597 13961 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:06.580003 13961 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:06.585660 13709 catalog_manager.cc:5582] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 6 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 7 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:06.629602 13809 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:06.630991 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:06.632711 13954 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:06.673193 15911 raft_consensus.cc:991] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:06.673812 15911 raft_consensus.cc:1079] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:06.686125 13960 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:06.686825 13960 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:06.702697 13960 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:06.662285 13961 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.713135 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:06.716948 13809 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:06.719498 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:06.723706 13885 raft_consensus.cc:3053] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:06.745476 13885 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:06.758458 13885 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.759793 13885 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:06.744872 13961 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:06.778782 13954 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:06.790180 15749 raft_consensus.cc:991] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:06.790910 15665 raft_consensus.cc:1079] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:06.791612 13809 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:06.792855 13926 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:06.793723 15782 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:06.794235 15782 raft_consensus.cc:695] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:06.795126 15782 consensus_queue.cc:237] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.801539 13809 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:06.802140 13809 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:06.802498 13809 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:06.810653 13809 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.814189 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:06.814790 13882 raft_consensus.cc:3053] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:06.815068 13882 raft_consensus.cc:738] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:06.815747 13882 consensus_queue.cc:260] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.816833 13882 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:06.817919 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:06.818501 13957 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:06.819010 13809 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:06.846787 13957 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:06.854082 13882 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:06.855662 13776 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:06.858110 15430 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:06.859397 15430 raft_consensus.cc:695] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:06.860394 15430 consensus_queue.cc:237] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.866727 13960 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.868566 13960 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:06.872359 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:06.872953 13882 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:06.873229 13882 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:06.874022 13882 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.877458 13882 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:06.877971 13709 catalog_manager.cc:5582] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 2 to 3, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 3 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:06.881800 13885 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:06.884944 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:06.908419 13809 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:06.886325 13882 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:06.910537 13925 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:06.911432 15782 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:06.911896 15782 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:06.912652 15782 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:06.949107 15719 raft_consensus.cc:991] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:06.950984 15719 raft_consensus.cc:1079] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:06.952700 13810 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:06.953347 13810 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:06.953790 13810 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:07.001493 13957 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:07.005349 15430 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:06.980818 13709 catalog_manager.cc:5582] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:07.011963 13809 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:07.028647 13810 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:07.031711 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:07.032253 13957 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:07.044075 13882 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:07.048843 15851 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:07.051354 13957 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:07.053278 13779 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:07.057459 15851 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:07.057971 15851 raft_consensus.cc:695] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:07.058868 15851 consensus_queue.cc:237] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:07.126214 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:07.126962 13885 raft_consensus.cc:3053] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:07.127319 13885 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:07.129007 13709 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:07.130530 13885 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:07.131759 13885 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:07.142349 13810 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:07.153200 13885 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:07.155791 13711 catalog_manager.cc:5582] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 5 to 6, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:07.338234 13882 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:07.357134 15782 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:07.363179 13810 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:07.364697 15782 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:07.439641 13886 raft_consensus.cc:1273] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:07.441067 15430 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:07.445600 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:07.445928 13882 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:07.459619 15782 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:07.494895 13961 raft_consensus.cc:1273] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:07.496372 15430 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:07.507956 13806 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:07.510025 15910 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:07.522735 13885 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:07.525087 15567 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:07.558882 13960 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:07.560585 15903 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:07.653612 13954 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:07.654284 13954 raft_consensus.cc:604] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:07.766844 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 125174us
I20250623 01:59:07.788676 13810 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:07.789367 13810 raft_consensus.cc:604] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:07.800624 13886 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:07.801381 13886 raft_consensus.cc:604] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:07.802670 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:07.804224 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:07.804900 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250623 01:59:07.870167 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 52761.2us
I20250623 01:59:07.878010 15933 raft_consensus.cc:991] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:07.878582 15933 raft_consensus.cc:1079] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:08.013103 15940 raft_consensus.cc:991] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:08.013638 15940 raft_consensus.cc:1079] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:08.023015 13961 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:08.027593 13961 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:08.029239 13961 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:59:08.051806 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 53805us
I20250623 01:59:08.067813 13809 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48570
I20250623 01:59:08.068795 13809 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:08.069178 13809 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:08.157608 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 74739.8us
I20250623 01:59:08.170646 13961 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:08.223119 13961 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:08.180804 13809 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:08.252238 13811 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:08.260478 13809 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:08.262466 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:08.263082 13885 raft_consensus.cc:3053] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:08.263375 13885 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:08.264086 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:08.264043 13885 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:08.264747 13886 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:08.265457 13885 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:08.272818 13886 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:08.281316 13776 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:08.284572 15944 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:08.285079 15944 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:08.285938 15944 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:08.287765 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:08.288378 13811 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:08.296552 13811 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:08.299517 13709 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 7 to 8, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 8 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:08.316566 15906 consensus_queue.cc:1046] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:08.317401 15906 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:08.317752 15906 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:08.318466 15906 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:08.319933 15906 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:08.320801 13885 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:08.325754 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:08.331111 13926 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:08.332624 15910 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:08.333793 15910 raft_consensus.cc:695] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:08.334885 15910 consensus_queue.cc:237] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:08.379673 13961 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:08.402216 13709 catalog_manager.cc:5582] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:08.414518 15944 raft_consensus.cc:991] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:08.415014 15944 raft_consensus.cc:1079] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:08.442816 13961 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37424
I20250623 01:59:08.443436 13961 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:08.443828 13961 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:08.455080 13961 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:08.460968 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:08.462306 13811 raft_consensus.cc:3053] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:08.462627 13811 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:08.463307 13811 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:08.464404 13811 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:08.474797 13961 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:08.475844 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:08.476402 13885 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:08.522047 13885 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:08.533571 13925 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:08.534438 15906 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:08.534992 15906 raft_consensus.cc:695] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:08.535181 13811 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:08.536028 15906 consensus_queue.cc:237] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:08.551817 13709 catalog_manager.cc:5582] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:08.802179 13809 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:08.803900 15910 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:08.852885 13882 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:08.854775 15903 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:08.881577 13961 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:08.883044 15903 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:08.943791 13881 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:08.948659 15910 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:08.954495 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:08.957897 13886 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:08.959306 15910 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250623 01:59:09.008284 15951 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.130s user 0.001s sys 0.000s
I20250623 01:59:09.070947 13957 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:09.071508 13957 raft_consensus.cc:604] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:09.073935 13806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:09.074447 13806 raft_consensus.cc:604] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:09.076941 13957 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c708cd61006343f09750b88b2621eec6"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:09.090718 13957 raft_consensus.cc:604] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:09.092361 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:09.094074 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:09.094836 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:09.137770 15941 raft_consensus.cc:991] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:09.138284 15941 raft_consensus.cc:1079] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:09.139801 13881 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:09.140307 13881 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:09.140668 13881 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:09.147151 13809 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:09.153542 15906 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:09.155882 13881 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:09.159989 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:09.160516 13809 raft_consensus.cc:3053] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:09.162214 13809 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:09.162982 13809 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:09.164039 13809 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:09.174978 13881 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:09.175143 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:09.175762 13954 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:09.188539 13954 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:09.194890 13854 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:09.195925 15943 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:09.196431 15943 raft_consensus.cc:695] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:09.197254 15943 consensus_queue.cc:237] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:09.227056 13809 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:09.275257 13711 catalog_manager.cc:5582] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 8, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:09.461982 15910 raft_consensus.cc:991] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:09.462499 15910 raft_consensus.cc:1079] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:09.464201 13881 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c708cd61006343f09750b88b2621eec6"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58122
I20250623 01:59:09.464831 13881 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:09.465190 13881 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:09.471796 13881 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:09.475711 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:09.476275 13809 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:09.476500 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:09.477114 13954 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:09.477452 13954 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:09.478206 13954 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:09.479285 13954 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:09.486274 13881 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:09.496008 13809 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:09.497357 13852 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:09.497910 13954 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:09.498214 15942 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:09.498790 15942 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:09.499711 15942 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:09.579821 13711 catalog_manager.cc:5582] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 5, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:09.602741 15906 raft_consensus.cc:991] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:09.603292 15906 raft_consensus.cc:1079] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:09.610831 13809 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:09.611248 13806 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48570
I20250623 01:59:09.611869 13806 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:09.612208 13806 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:09.613619 15942 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:09.622823 13806 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:09.627202 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:09.627776 13881 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:09.637900 13881 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:09.654211 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:09.654830 14508 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:09.658039 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:09.658826 13954 raft_consensus.cc:3053] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:09.654336 13806 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:09.660351 13954 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:09.673962 13960 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:09.675266 15970 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:09.675927 14508 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:09.677052 13778 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:09.677878 15903 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:09.678380 15903 raft_consensus.cc:695] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:09.679203 15903 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:09.686395 13954 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:09.687716 13954 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:09.689203 13881 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:09.704561 13954 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:09.796859 13711 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 5 to 6, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:09.933094 13957 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:09.940320 15970 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:10.026345 13806 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:10.027907 15970 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:10.059242 13881 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:10.061393 15975 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:10.092337 14506 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:10.099253 15975 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:10.112759 13957 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:10.114216 15976 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:10.188295 14508 consensus_queue.cc:237] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:10.233965 13806 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250623 01:59:10.235448 15909 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:10.238539 13881 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250623 01:59:10.248390 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:10.254207 15909 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:59:10.281795 15950 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:10.283303 13811 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:10.293429 13756 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 272c03e6252d489cbf2fdceb36b05dc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 11)
I20250623 01:59:10.306967 13711 catalog_manager.cc:5582] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index 3 to 6, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New cstate: current_term: 5 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:10.326400 13881 raft_consensus.cc:2953] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:10.359047 13951 tablet_service.cc:1515] Processing DeleteTablet for tablet 272c03e6252d489cbf2fdceb36b05dc1 with delete_type TABLET_DATA_TOMBSTONED (TS c243715fe12445019c33bbeafbaea635 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37408
I20250623 01:59:10.389098 13885 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:10.389812 13885 raft_consensus.cc:604] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:10.395251 15991 tablet_replica.cc:331] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: stopping tablet replica
I20250623 01:59:10.396265 15991 raft_consensus.cc:2241] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:10.399673 13811 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:10.400383 13811 raft_consensus.cc:604] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:10.397105 15991 raft_consensus.cc:2270] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Raft consensus is shut down!
W20250623 01:59:10.404363 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 76972.3us
I20250623 01:59:10.410761 13881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e773c5700b0945d1981f481534952e85"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:10.415907 13881 raft_consensus.cc:604] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:10.422216 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:10.423458 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:10.424234 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:10.434671 15991 ts_tablet_manager.cc:1905] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:10.511469 15991 ts_tablet_manager.cc:1918] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.5
I20250623 01:59:10.511907 15991 log.cc:1199] T 272c03e6252d489cbf2fdceb36b05dc1 P c243715fe12445019c33bbeafbaea635: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/wals/272c03e6252d489cbf2fdceb36b05dc1
I20250623 01:59:10.517414 13697 catalog_manager.cc:4928] TS c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): tablet 272c03e6252d489cbf2fdceb36b05dc1 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:10.604176 15969 raft_consensus.cc:991] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:10.604840 15963 raft_consensus.cc:1079] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:10.606408 13811 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e773c5700b0945d1981f481534952e85"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:10.606900 13811 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:10.607224 13811 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:10.615571 13811 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.620488 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:10.620997 13885 raft_consensus.cc:3053] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:10.621265 13885 raft_consensus.cc:738] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:10.621974 13885 consensus_queue.cc:260] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.622974 13885 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:10.631783 13885 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:10.632750 13776 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:10.632838 13811 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:10.633530 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:10.633720 15976 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:10.634178 13961 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:10.634320 15976 raft_consensus.cc:695] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:10.635193 15976 consensus_queue.cc:237] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.650864 13961 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:10.685355 13711 catalog_manager.cc:5582] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:10.709493 15969 raft_consensus.cc:991] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:10.709993 15969 raft_consensus.cc:1079] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:10.712971 13961 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:10.714201 13961 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:10.714525 13961 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:10.726460 13961 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.729727 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:10.730304 13885 raft_consensus.cc:3053] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:10.730731 13885 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:10.731380 13885 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.732313 13885 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:10.737823 15976 raft_consensus.cc:991] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:10.738329 15976 raft_consensus.cc:1079] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:10.740154 13957 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37424
I20250623 01:59:10.740676 13957 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:10.741004 13957 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:10.729734 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:10.744099 13806 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:10.749891 13957 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.750553 13806 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:10.751307 13961 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:10.751595 13926 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:10.752380 15906 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:10.752844 15906 raft_consensus.cc:695] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:10.753623 15906 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.754037 13885 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:10.777136 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:10.777846 13806 raft_consensus.cc:3053] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:10.778183 13806 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:10.778889 13806 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.779991 13806 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:10.783139 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:10.783691 13885 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:10.785964 13957 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:10.799376 13711 catalog_manager.cc:5582] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 5 to 6, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 6 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:10.810398 13885 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:10.812273 13925 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:10.816270 15906 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:10.816780 15906 raft_consensus.cc:695] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:10.817734 13806 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:10.817524 15906 consensus_queue.cc:237] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:10.911376 13711 catalog_manager.cc:5582] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:11.198652 13957 raft_consensus.cc:1273] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:11.241776 13881 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:11.243273 15997 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:11.259231 13882 raft_consensus.cc:1273] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:11.261065 15959 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:11.312081 15976 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:11.312883 13881 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:11.325712 15933 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:11.431363 13811 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:11.511909 15910 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:11.554221 13811 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:11.592235 15910 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:11.630241 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:11.879670 13806 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:11.943658 14508 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:11.945508 15959 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:11.977891 13961 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:11.992844 15959 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250623 01:59:12.015250 16012 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20250623 01:59:12.016494 15959 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 3 to 7, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:12.020471 13960 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:12.021046 13960 raft_consensus.cc:604] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:12.028963 13811 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:12.029841 13811 raft_consensus.cc:604] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:12.033998 13960 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:12.034875 13960 raft_consensus.cc:604] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:12.035982 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:12.037671 14508 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 3 to 7, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:12.054774 13711 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index 3 to 7, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:12.057992 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:12.058655 16009 raft_consensus.cc:991] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:12.059121 16009 raft_consensus.cc:1079] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:12.059811 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:12.061180 13881 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58122
I20250623 01:59:12.061765 13881 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:12.062091 13881 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:12.069075 13881 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:12.071381 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:12.071998 13960 raft_consensus.cc:3053] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:12.072328 13960 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:12.072912 13881 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:59:12.073063 13960 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:12.074393 13960 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:12.074277 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:12.074962 13811 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:12.081166 13811 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:12.081512 13695 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0f12346da5a14097bbfa150cb8d1d907 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 11)
I20250623 01:59:12.082383 13852 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:12.083164 15963 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:12.083950 15963 raft_consensus.cc:695] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:12.084748 15963 consensus_queue.cc:237] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:12.081166 13960 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:12.126531 13961 raft_consensus.cc:2953] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 3 to 7, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:12.136564 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f12346da5a14097bbfa150cb8d1d907 with delete_type TABLET_DATA_TOMBSTONED (TS ee041a8325864e788a383b8356afc652 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58120
I20250623 01:59:12.150518 16009 raft_consensus.cc:991] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:12.151082 16009 raft_consensus.cc:1079] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:12.158931 13811 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48570
I20250623 01:59:12.159458 13811 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:12.159816 13811 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:12.204123 16021 tablet_replica.cc:331] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: stopping tablet replica
I20250623 01:59:12.217484 16021 raft_consensus.cc:2241] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:12.218153 13708 catalog_manager.cc:5582] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 8, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:12.207721 13811 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:12.233914 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:12.234496 13881 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:12.237809 13811 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:12.243359 13881 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:12.245374 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:12.245934 13961 raft_consensus.cc:3053] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:12.246212 13961 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:12.246948 13961 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:12.247642 13776 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:12.254408 15959 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:12.254521 13961 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:12.255657 15959 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:12.256639 15959 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:12.251072 16021 raft_consensus.cc:2270] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:12.266391 16021 ts_tablet_manager.cc:1905] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:12.270771 13961 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:12.303227 13708 catalog_manager.cc:5582] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 5, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:12.329033 16021 ts_tablet_manager.cc:1918] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.6
I20250623 01:59:12.329632 16021 log.cc:1199] T 0f12346da5a14097bbfa150cb8d1d907 P ee041a8325864e788a383b8356afc652: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/wals/0f12346da5a14097bbfa150cb8d1d907
I20250623 01:59:12.361555 13694 catalog_manager.cc:4928] TS ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): tablet 0f12346da5a14097bbfa150cb8d1d907 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:12.533905 15959 raft_consensus.cc:991] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:12.534428 15959 raft_consensus.cc:1079] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:12.536026 13881 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:12.536614 13881 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:12.536969 13881 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:12.601557 13811 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:12.602257 13881 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:12.604856 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:12.605412 13806 raft_consensus.cc:3053] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:12.605690 13806 raft_consensus.cc:738] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:12.606349 13806 consensus_queue.cc:260] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:12.606884 15970 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:12.607318 13806 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:12.615500 13806 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:12.627382 13852 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:12.633783 16022 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:12.634271 16022 raft_consensus.cc:695] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:12.635085 16022 consensus_queue.cc:237] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:12.641747 13961 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:12.643386 16023 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:12.674108 13708 catalog_manager.cc:5582] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 3, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 3 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:12.677546 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:12.678099 13960 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:12.672466 13881 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:12.696465 13960 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:12.775884 13881 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:12.777354 15971 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:12.814002 13960 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:12.818035 15959 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:13.028385 13960 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:13.029968 16023 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:13.061149 13811 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:13.078771 15970 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:13.272045 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:13.433162 13812 consensus_queue.cc:237] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:13.504086 13813 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:13.504904 13813 raft_consensus.cc:604] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:13.513983 13882 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c708cd61006343f09750b88b2621eec6"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:13.516346 13882 raft_consensus.cc:604] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:13.523476 13882 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:13.524550 13882 raft_consensus.cc:604] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:13.526209 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:13.528080 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:13.529183 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:13.551249 13960 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20250623 01:59:13.552742 13778 consensus_peers.cc:487] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92909e3375c549b5baa0595e1750fa1f. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:13.552814 15971 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:13.555115 13882 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:13.556511 15971 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:13.600282 15971 raft_consensus.cc:991] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:13.601066 15959 raft_consensus.cc:1079] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:13.602840 13961 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37424
I20250623 01:59:13.603356 13961 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:13.603711 13961 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:13.614140 15959 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:13.618778 13961 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:13.622033 13961 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:13.625032 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:13.625829 13884 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:13.645817 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:13.646438 13813 raft_consensus.cc:3053] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:13.646749 13813 raft_consensus.cc:738] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:13.647464 13813 consensus_queue.cc:260] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:13.648682 13884 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:13.648747 13813 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:13.648398 13960 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:13.659687 13813 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:13.663105 13925 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:13.702576 16037 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:13.703028 16037 raft_consensus.cc:695] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:13.703945 16037 consensus_queue.cc:237] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:13.649401 13886 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:13.686471 13711 catalog_manager.cc:5582] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index -1 to 4, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:13.699213 15970 raft_consensus.cc:991] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:13.712651 15970 raft_consensus.cc:1079] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:13.714406 13806 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c708cd61006343f09750b88b2621eec6"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:13.714915 13806 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:13.715232 13806 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:13.723232 15963 raft_consensus.cc:991] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:13.723872 16039 raft_consensus.cc:1079] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:13.699097 13884 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:13.726336 13806 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:13.728389 13810 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:13.729111 13810 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:13.730423 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:13.733898 13810 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:13.734077 13885 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:13.734369 13885 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:13.735137 13885 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:13.729065 13806 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:13.741673 13885 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:13.747982 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:13.748572 13960 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:13.757702 13885 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:13.764508 13810 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:13.770635 13776 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:13.772449 16043 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:13.772948 16043 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:13.773811 16043 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:13.797984 13810 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:13.800124 13960 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:13.800366 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:13.802654 13886 raft_consensus.cc:3053] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:59:13.803061 13886 raft_consensus.cc:738] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:13.806087 13886 consensus_queue.cc:260] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:13.808627 13886 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:13.807459 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:13.809417 13954 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:13.812196 13961 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:13.817521 13886 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:13.831132 15963 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250623 01:59:13.833171 13853 consensus_peers.cc:487] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ac15d89950f4485a84912743c361f1d. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:13.834393 13776 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:13.834578 13813 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:13.836361 16039 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:13.840163 15959 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:59:13.840792 15959 raft_consensus.cc:695] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:13.841642 15959 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:13.890275 16048 log.cc:927] Time spent T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250623 01:59:13.897419 15963 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:13.904562 13961 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:13.912384 13954 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:13.932832 13813 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:13.991873 13885 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:13.993333 13710 catalog_manager.cc:5582] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:13.995050 13710 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index -1 to 5, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:14.005141 13711 catalog_manager.cc:5582] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 5 to 6, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:14.007090 13711 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
W20250623 01:59:14.036070 13853 consensus_peers.cc:487] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 819094ffb4314769ab2a06512232fc61. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:14.041471 13961 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:14.045199 16039 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:14.066646 13813 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:14.068363 15963 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:14.108577 15970 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:14.111626 16057 ts_tablet_manager.cc:927] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:59:14.120628 13813 raft_consensus.cc:1273] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:14.123076 16057 tablet_copy_client.cc:323] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.1:34781
I20250623 01:59:14.125833 13823 tablet_copy_service.cc:140] P 85a58a834b6b4b0dbe197b3dad87e382: Received BeginTabletCopySession request for tablet 92909e3375c549b5baa0595e1750fa1f from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:41526)
I20250623 01:59:14.126394 13823 tablet_copy_service.cc:161] P 85a58a834b6b4b0dbe197b3dad87e382: Beginning new tablet copy session on tablet 92909e3375c549b5baa0595e1750fa1f from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:41526: session id = 7d22741cfb9b408684cb82c58241b2f5-92909e3375c549b5baa0595e1750fa1f
I20250623 01:59:14.126753 15906 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:14.135632 13823 tablet_copy_source_session.cc:215] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:59:14.148736 16057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92909e3375c549b5baa0595e1750fa1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:14.169046 13960 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:14.172019 15959 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:14.191761 16057 tablet_copy_client.cc:806] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:59:14.192451 16057 tablet_copy_client.cc:670] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:59:14.188498 13813 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:14.234068 13884 raft_consensus.cc:1273] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:14.235585 15906 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:14.255468 16057 tablet_copy_client.cc:538] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250623 01:59:14.256786 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 48484.3us
I20250623 01:59:14.270256 16057 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:59:14.323451 13711 catalog_manager.cc:5582] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index -1 to 5, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:14.337616 13886 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:14.340929 13812 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:14.352931 13954 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:14.355801 16043 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
W20250623 01:59:14.473479 16059 log.cc:927] Time spent T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
W20250623 01:59:14.483052 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 67730.9us
W20250623 01:59:14.485473 16047 log.cc:927] Time spent T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250623 01:59:14.561740 13884 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:59:14.633755 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250623 01:59:14.662155 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 169261us
I20250623 01:59:14.737555 13960 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20250623 01:59:14.736620 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 178885us
W20250623 01:59:14.790161 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 119199us
W20250623 01:59:14.859802 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 105833us
I20250623 01:59:14.918740 13953 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250623 01:59:14.978127 16079 log.cc:927] Time spent T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
W20250623 01:59:14.989509 16060 log.cc:927] Time spent T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20250623 01:59:15.076876 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 126168us
I20250623 01:59:15.093590 16098 ts_tablet_manager.cc:927] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:15.097529 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 282651us
I20250623 01:59:15.120534 16098 tablet_copy_client.cc:323] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.2:43055
I20250623 01:59:15.121896 16085 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250623 01:59:15.089092 16043 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20250623 01:59:15.157399 16110 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:59:15.157927 16057 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:15.158969 16057 tablet_bootstrap.cc:492] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:59:15.160179 13881 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:15.160300 16057 ts_tablet_manager.cc:1397] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.890s user 0.040s sys 0.000s
W20250623 01:59:15.146543 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 103664us
W20250623 01:59:15.166685 13778 consensus_peers.cc:487] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b3574beac134b2eabbd65a4f54b3ab3. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:15.201109 13885 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.004s sys 0.008s
W20250623 01:59:15.201683 13885 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.510s user 0.004s sys 0.008s
I20250623 01:59:15.203217 13898 tablet_copy_service.cc:140] P ee041a8325864e788a383b8356afc652: Received BeginTabletCopySession request for tablet 6ac15d89950f4485a84912743c361f1d from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:59310)
I20250623 01:59:15.203787 13898 tablet_copy_service.cc:161] P ee041a8325864e788a383b8356afc652: Beginning new tablet copy session on tablet 6ac15d89950f4485a84912743c361f1d from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:59310: session id = 7d22741cfb9b408684cb82c58241b2f5-6ac15d89950f4485a84912743c361f1d
W20250623 01:59:15.224112 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 61965.2us
I20250623 01:59:15.224949 16097 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:15.227267 13898 tablet_copy_source_session.cc:215] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:59:15.233182 16098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ac15d89950f4485a84912743c361f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:15.243038 16114 ts_tablet_manager.cc:927] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:15.244532 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 79552.6us
I20250623 01:59:15.246060 16057 raft_consensus.cc:357] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:15.246939 16057 raft_consensus.cc:738] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:59:15.247635 16057 consensus_queue.cc:260] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
W20250623 01:59:15.371168 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 82667.9us
I20250623 01:59:15.395826 16098 tablet_copy_client.cc:806] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:59:15.412425 16098 tablet_copy_client.cc:670] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
W20250623 01:59:15.430097 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 155719us
I20250623 01:59:15.418267 16114 tablet_copy_client.cc:323] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.2:43055
I20250623 01:59:15.433702 16098 tablet_copy_client.cc:538] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:59:15.434478 13898 tablet_copy_service.cc:140] P ee041a8325864e788a383b8356afc652: Received BeginTabletCopySession request for tablet 819094ffb4314769ab2a06512232fc61 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:59310)
I20250623 01:59:15.435046 13898 tablet_copy_service.cc:161] P ee041a8325864e788a383b8356afc652: Beginning new tablet copy session on tablet 819094ffb4314769ab2a06512232fc61 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:59310: session id = 7d22741cfb9b408684cb82c58241b2f5-819094ffb4314769ab2a06512232fc61
I20250623 01:59:15.432540 13810 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:15.451207 13810 raft_consensus.cc:604] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:15.456104 16057 ts_tablet_manager.cc:1428] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.295s user 0.007s sys 0.000s
I20250623 01:59:15.444567 16098 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:59:15.444645 13898 tablet_copy_source_session.cc:215] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:59:15.465276 13953 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:15.467751 13953 raft_consensus.cc:604] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:15.469082 13823 tablet_copy_service.cc:342] P 85a58a834b6b4b0dbe197b3dad87e382: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-92909e3375c549b5baa0595e1750fa1f received from {username='slave'} at 127.0.0.1:41526
I20250623 01:59:15.469611 13823 tablet_copy_service.cc:434] P 85a58a834b6b4b0dbe197b3dad87e382: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-92909e3375c549b5baa0595e1750fa1f on tablet 92909e3375c549b5baa0595e1750fa1f with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:15.487349 13806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c708cd61006343f09750b88b2621eec6"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:15.487932 13806 raft_consensus.cc:604] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:15.508908 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:15.511229 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:15.512064 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:15.606215 16114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819094ffb4314769ab2a06512232fc61. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 01:59:15.626178 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 68846.5us
I20250623 01:59:15.630034 16102 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:15.653658 16057 ts_tablet_manager.cc:927] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:59:15.730007 14506 raft_consensus.cc:1215] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.3->[4.4-4.4] Dedup: 4.4->[]
I20250623 01:59:15.760980 16057 tablet_copy_client.cc:323] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.1:34781
I20250623 01:59:15.762574 13823 tablet_copy_service.cc:140] P 85a58a834b6b4b0dbe197b3dad87e382: Received BeginTabletCopySession request for tablet 1b3574beac134b2eabbd65a4f54b3ab3 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:41526)
I20250623 01:59:15.763103 13823 tablet_copy_service.cc:161] P 85a58a834b6b4b0dbe197b3dad87e382: Beginning new tablet copy session on tablet 1b3574beac134b2eabbd65a4f54b3ab3 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:41526: session id = 7d22741cfb9b408684cb82c58241b2f5-1b3574beac134b2eabbd65a4f54b3ab3
I20250623 01:59:15.765338 13885 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:15.777477 16094 raft_consensus.cc:991] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:15.778141 16075 raft_consensus.cc:1079] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:15.779789 16102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:41518 (request call id 5) took 1468 ms. Trace:
I20250623 01:59:15.780894 13881 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:15.780192 16102 rpcz_store.cc:276] 0623 01:59:14.311427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:14.312455 (+ 1028us) service_pool.cc:225] Handling call
0623 01:59:14.364181 (+ 51726us) log.cc:844] Serialized 492 byte log entry
0623 01:59:14.975396 (+611215us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b180043a140. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b4061f 00007f41e0b428c1 00007f41e0b42879 00007f41e0b42809 00007f41e0b427d1 00007f41e0b41acd 00007f41e9b9b124 00007f41e9b9af59 00007f41e0b23646
0623 01:59:14.975876 (+ 480us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b2c000014c8. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e24426d1 00007f41e245139a 00007f41e2451023 00007f41e2450fc1 00007f41e3adbda2 00007f41e3adbf45 00007f41e3a2fb51 00007f41e3a18372 00007f41e3a1d781 00007f41e3a1d709 00007f41e3a1d699
0623 01:59:15.779708 (+803832us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60632,"raft.queue_time_us":1514,"raft.run_cpu_time_us":3076,"raft.run_wall_time_us":437891,"spinlock_wait_cycles":11864576,"thread_start_us":98818,"threads_started":1,"wal-append.queue_time_us":400990}
I20250623 01:59:15.781711 13881 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:15.794580 16094 raft_consensus.cc:991] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:15.794685 13881 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:15.795189 16102 raft_consensus.cc:1079] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:15.802718 13884 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c708cd61006343f09750b88b2621eec6"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:15.803260 13884 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:15.803608 13884 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:15.824741 16107 raft_consensus.cc:991] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:15.833719 13881 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:15.842191 16126 log.cc:927] Time spent T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.115s user 0.003s sys 0.000s
I20250623 01:59:15.842362 13881 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:15.848551 13884 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:15.871959 13884 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:15.870069 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 112049us
I20250623 01:59:15.876017 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:15.834467 16107 raft_consensus.cc:1079] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:15.876636 13960 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:15.881883 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:15.882486 13959 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:15.898217 13823 tablet_copy_source_session.cc:215] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:59:15.898643 13959 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:15.899996 13954 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:15.901521 16057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b3574beac134b2eabbd65a4f54b3ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:15.904721 16114 tablet_copy_client.cc:806] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:59:15.905274 16114 tablet_copy_client.cc:670] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:59:15.908741 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:15.909265 13812 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:15.909610 13812 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:15.908967 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:15.910272 13812 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:15.910676 13813 raft_consensus.cc:3053] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:15.911319 13812 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:15.911388 13813 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:15.912281 13854 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:15.915305 16101 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:15.915750 16101 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:15.916770 16114 tablet_copy_client.cc:538] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:59:15.916510 16101 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:15.923746 13710 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index -1 to 7, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 8 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:15.930924 13811 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48570
I20250623 01:59:15.930505 13953 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:15.931449 13811 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:15.934522 13811 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:15.939281 13813 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:15.940711 13813 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:15.947077 13811 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:15.948871 13811 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:15.949671 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:15.950208 13882 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:15.957849 13882 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:15.960507 13960 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:15.962520 13854 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:15.964144 16089 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:15.964639 16089 raft_consensus.cc:695] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:15.967104 16089 consensus_queue.cc:237] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:15.971895 16114 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:59:15.961100 13813 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:15.972258 13776 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:15.973554 13812 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:15.975335 13811 raft_consensus.cc:1238] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:15.977265 16132 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:15.977749 16132 raft_consensus.cc:695] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:15.978598 16132 consensus_queue.cc:237] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:15.989970 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:15.990597 13960 raft_consensus.cc:3053] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:15.990880 13960 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:16.010641 16131 consensus_queue.cc:1046] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20250623 01:59:16.025490 13960 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:16.026721 13960 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:16.033380 13960 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
W20250623 01:59:16.033825 16131 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20250623 01:59:16.036178 16057 tablet_copy_client.cc:806] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:59:16.082325 16098 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:16.093631 16057 tablet_copy_client.cc:670] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:59:16.102609 16057 tablet_copy_client.cc:538] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:59:16.109078 16098 tablet_bootstrap.cc:492] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:59:16.119151 16098 ts_tablet_manager.cc:1397] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.675s user 0.027s sys 0.016s
I20250623 01:59:16.136511 16098 raft_consensus.cc:357] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:16.137358 16098 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:59:16.138001 16098 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:16.140645 16057 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:59:16.186051 16098 ts_tablet_manager.cc:1428] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20250623 01:59:16.189026 13898 tablet_copy_service.cc:342] P ee041a8325864e788a383b8356afc652: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-6ac15d89950f4485a84912743c361f1d received from {username='slave'} at 127.0.0.1:59310
I20250623 01:59:16.189495 13898 tablet_copy_service.cc:434] P ee041a8325864e788a383b8356afc652: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-6ac15d89950f4485a84912743c361f1d on tablet 6ac15d89950f4485a84912743c361f1d with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:16.185927 13709 catalog_manager.cc:5582] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 7, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:16.195401 13709 catalog_manager.cc:5582] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:16.217031 13708 catalog_manager.cc:5582] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 5, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:16.222791 16073 raft_consensus.cc:1062] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:59:16.225818 16073 consensus_queue.cc:237] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:16.260640 16114 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=1 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:16.270042 13953 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250623 01:59:16.272233 13881 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250623 01:59:16.272409 16114 tablet_bootstrap.cc:492] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:59:16.277886 16114 ts_tablet_manager.cc:1397] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.306s user 0.033s sys 0.010s
I20250623 01:59:16.273855 16108 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250623 01:59:16.273097 14506 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEARNER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250623 01:59:16.279170 16073 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:16.280292 16108 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:16.314955 16073 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:16.316224 13953 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:16.325553 13813 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:16.328485 16089 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20250623 01:59:16.339778 16114 raft_consensus.cc:357] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:16.341550 16114 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:59:16.351215 13708 catalog_manager.cc:5582] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:16.355739 16114 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:16.370389 14506 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:16.371506 13882 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:16.444679 14508 raft_consensus.cc:1215] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.4->[7.5-7.5] Dedup: 7.5->[]
I20250623 01:59:16.455374 16114 ts_tablet_manager.cc:1428] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.177s user 0.007s sys 0.002s
I20250623 01:59:16.464047 13809 consensus_queue.cc:237] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:16.478040 13953 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:16.482091 13898 tablet_copy_service.cc:342] P ee041a8325864e788a383b8356afc652: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-819094ffb4314769ab2a06512232fc61 received from {username='slave'} at 127.0.0.1:59310
I20250623 01:59:16.482578 13898 tablet_copy_service.cc:434] P ee041a8325864e788a383b8356afc652: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-819094ffb4314769ab2a06512232fc61 on tablet 819094ffb4314769ab2a06512232fc61 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:16.486586 13962 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:59:16.484324 13882 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:16.493645 16073 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:16.495801 13953 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:16.497036 16109 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:16.497480 16101 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:16.507958 13809 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250623 01:59:16.510272 16057 log.cc:927] Time spent T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.057s user 0.004s sys 0.000s
I20250623 01:59:16.512806 16134 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:16.514971 16073 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:16.516840 13962 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:16.518410 16057 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:16.529959 16134 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:16.554116 16057 tablet_bootstrap.cc:492] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:59:16.554942 16057 ts_tablet_manager.cc:1397] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.415s user 0.033s sys 0.023s
I20250623 01:59:16.576089 14507 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:59:16.602728 16109 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:16.605376 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:16.616441 13954 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:16.644254 16057 raft_consensus.cc:357] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
W20250623 01:59:16.680521 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 136143us
I20250623 01:59:16.677517 16057 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:59:16.625401 16108 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20250623 01:59:16.769735 16057 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:16.802000 14507 raft_consensus.cc:1215] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Deduplicated request from leader. Original: 6.4->[6.5-6.5] Dedup: 6.5->[]
I20250623 01:59:16.803277 16057 ts_tablet_manager.cc:1428] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.248s user 0.005s sys 0.004s
W20250623 01:59:16.877130 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 154050us
I20250623 01:59:16.898147 13823 tablet_copy_service.cc:342] P 85a58a834b6b4b0dbe197b3dad87e382: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-1b3574beac134b2eabbd65a4f54b3ab3 received from {username='slave'} at 127.0.0.1:41526
I20250623 01:59:16.908892 13823 tablet_copy_service.cc:434] P 85a58a834b6b4b0dbe197b3dad87e382: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-1b3574beac134b2eabbd65a4f54b3ab3 on tablet 1b3574beac134b2eabbd65a4f54b3ab3 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:16.938043 14507 raft_consensus.cc:2953] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:17.017153 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 53707.3us
W20250623 01:59:17.012848 16155 log.cc:927] Time spent T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20250623 01:59:17.058362 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 109486us
W20250623 01:59:17.025432 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 64115.3us
W20250623 01:59:17.081995 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 143258us
I20250623 01:59:17.102084 13695 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 92909e3375c549b5baa0595e1750fa1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:59:17.134018 13884 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c708cd61006343f09750b88b2621eec6"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:17.134727 13884 raft_consensus.cc:604] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:17.156056 13882 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:17.156631 13882 raft_consensus.cc:604] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:17.160708 13813 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:17.161224 13813 raft_consensus.cc:604] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:17.163512 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:17.166877 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:17.168968 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:17.179839 16134 raft_consensus.cc:991] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:17.180310 16134 raft_consensus.cc:1079] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:17.183637 13813 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c708cd61006343f09750b88b2621eec6"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:17.184363 13813 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:17.184772 13813 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:17.186164 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 102025us
I20250623 01:59:17.192781 13813 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.199944 13813 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:17.208041 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 264860us
I20250623 01:59:17.214912 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:17.215513 13885 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:17.215787 13885 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:17.216461 13885 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.217478 13885 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:17.212822 13708 catalog_manager.cc:5582] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index 5 to 6, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:17.241636 13885 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:17.249466 14506 raft_consensus.cc:1215] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEARNER]: Deduplicated request from leader. Original: 8.6->[8.7-8.7] Dedup: 8.7->[]
I20250623 01:59:17.252261 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:17.252938 13962 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:17.260152 13962 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:17.265410 13776 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:17.266418 16185 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:17.266923 16185 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:17.267771 16185 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.273622 16166 raft_consensus.cc:991] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:17.274098 16166 raft_consensus.cc:1079] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:17.278060 13960 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37424
I20250623 01:59:17.278575 13960 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:17.278915 13960 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:17.290002 13960 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.295522 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:17.296097 13813 raft_consensus.cc:3053] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:17.296382 13813 raft_consensus.cc:738] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:17.296522 16089 raft_consensus.cc:991] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:17.297170 16134 raft_consensus.cc:1079] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:17.297070 13813 consensus_queue.cc:260] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.298120 13813 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:17.300647 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:17.301394 13960 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:17.301528 13809 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:17.303232 13809 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:17.303588 13809 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:17.305002 13813 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:17.322947 16089 raft_consensus.cc:1062] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:59:17.325083 16089 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 3, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:17.347609 13809 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.356201 16157 raft_consensus.cc:1062] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:59:17.358661 16157 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 3, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:17.361033 13813 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:17.363869 13926 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:17.365098 13953 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:17.370524 16162 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:17.371305 16162 raft_consensus.cc:695] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:17.372244 16162 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.375242 13809 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:17.375834 16134 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:17.365618 13884 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:17.409853 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:17.410518 13879 raft_consensus.cc:3053] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:17.410827 13879 raft_consensus.cc:738] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:17.411590 13879 consensus_queue.cc:260] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.412564 13879 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:17.419229 14509 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEARNER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:17.424839 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:17.425490 13953 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:17.429883 13879 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:17.432873 13776 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:17.438118 16185 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:17.438715 16185 raft_consensus.cc:695] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:17.439716 16185 consensus_queue.cc:237] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:17.445992 13953 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
W20250623 01:59:17.433259 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:17.449198 14509 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20250623 01:59:17.449148 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:17.434409 13809 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:17.450677 16089 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:17.456259 13960 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:17.444557 16134 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20250623 01:59:17.474346 13884 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:17.474999 13881 raft_consensus.cc:1238] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:59:17.477047 13960 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:17.481216 16181 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:17.478243 16134 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250623 01:59:17.499346 16181 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20250623 01:59:17.566555 14509 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.559211 16134 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.590801 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet 92909e3375c549b5baa0595e1750fa1f with delete_type TABLET_DATA_TOMBSTONED (TS ee041a8325864e788a383b8356afc652 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58120
I20250623 01:59:17.618211 13710 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.619513 13709 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 3, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 3 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:17.619812 14509 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.617187 13711 catalog_manager.cc:5582] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 7 to 8, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 8 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:17.609102 16181 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.625836 13711 catalog_manager.cc:5582] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:17.673362 13960 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.691677 16185 raft_consensus.cc:1062] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:59:17.694427 13708 catalog_manager.cc:5582] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:17.687494 13809 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.694415 16185 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:17.718025 13962 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.724700 13809 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.730315 14508 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEARNER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:17.734661 16109 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:17.748560 13962 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:17.823730 16204 tablet_replica.cc:331] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: stopping tablet replica
I20250623 01:59:17.840991 13879 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:17.869760 13882 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:17.874384 16175 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250623 01:59:17.877646 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 128935us
I20250623 01:59:17.865093 13879 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:17.880005 13778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 44526.8us
I20250623 01:59:17.880182 16157 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:17.880697 16157 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:17.883157 16157 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:17.884235 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:17.887032 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:17.888650 13852 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:17.888671 14508 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:17.889663 16181 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:17.890627 16157 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:17.891115 13806 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:17.891309 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:17.892369 16089 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250623 01:59:17.898352 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 50438us
I20250623 01:59:17.905258 16166 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:17.906085 16204 raft_consensus.cc:2241] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:17.906884 16204 raft_consensus.cc:2270] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:17.908465 16159 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.918157 13883 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:17.918633 13806 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.924281 16166 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:17.927711 13809 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:17.929379 13960 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:17.930851 13694 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 819094ffb4314769ab2a06512232fc61 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:59:17.900022 14508 raft_consensus.cc:2953] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:17.944249 16176 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:17.948953 16132 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:17.950237 16204 ts_tablet_manager.cc:1905] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:17.971478 13879 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:17.984505 13711 catalog_manager.cc:5582] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 6 to 7, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:18.004236 13951 tablet_service.cc:1515] Processing DeleteTablet for tablet 819094ffb4314769ab2a06512232fc61 with delete_type TABLET_DATA_TOMBSTONED (TS c243715fe12445019c33bbeafbaea635 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37408
I20250623 01:59:17.994573 16185 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20250623 01:59:18.032148 13960 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250623 01:59:18.035208 13960 raft_consensus.cc:1273] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:18.038488 16214 tablet_replica.cc:331] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: stopping tablet replica
I20250623 01:59:18.039263 16214 raft_consensus.cc:2241] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:18.039908 16214 raft_consensus.cc:2270] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:18.048328 16108 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250623 01:59:18.055812 16214 ts_tablet_manager.cc:1905] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:18.043072 16159 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20250623 01:59:18.126685 13962 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:18.153121 13954 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:18.179970 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 187201us
I20250623 01:59:18.192796 13882 raft_consensus.cc:1273] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:18.104880 16173 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:18.257782 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:18.276098 14508 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:18.251938 16209 log.cc:927] Time spent T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
W20250623 01:59:18.281121 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 58148.3us
W20250623 01:59:18.372184 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 111665us
W20250623 01:59:18.344930 16215 log.cc:927] Time spent T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
W20250623 01:59:18.443140 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 68439.8us
I20250623 01:59:18.447160 13960 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:18.447945 13960 raft_consensus.cc:604] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:18.429790 14509 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250623 01:59:18.376019 16166 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:18.380394 16227 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
W20250623 01:59:18.476725 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 187829us
I20250623 01:59:18.574203 13888 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:18.514115 16181 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20250623 01:59:18.613039 16214 ts_tablet_manager.cc:1918] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.6
I20250623 01:59:18.613597 16214 log.cc:1199] T 819094ffb4314769ab2a06512232fc61 P c243715fe12445019c33bbeafbaea635: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/wals/819094ffb4314769ab2a06512232fc61
I20250623 01:59:18.615410 13697 catalog_manager.cc:4928] TS c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): tablet 819094ffb4314769ab2a06512232fc61 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:18.626834 13710 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index 6 to 7, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:18.651209 13711 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 8 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:18.658217 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 52688.9us
I20250623 01:59:18.653661 16204 ts_tablet_manager.cc:1918] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20250623 01:59:18.690119 16204 log.cc:1199] T 92909e3375c549b5baa0595e1750fa1f P ee041a8325864e788a383b8356afc652: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/wals/92909e3375c549b5baa0595e1750fa1f
I20250623 01:59:18.691601 16204 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58120 (request call id 546) took 1101 ms. Trace:
W20250623 01:59:18.702565 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 92872us
I20250623 01:59:18.744259 13811 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:18.744899 13811 raft_consensus.cc:604] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:18.758276 13694 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6ac15d89950f4485a84912743c361f1d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:59:18.785844 14508 raft_consensus.cc:2953] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:18.824345 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250623 01:59:18.828881 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:18.829967 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:18.692065 16204 rpcz_store.cc:276] 0623 01:59:17.589997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:17.590466 (+ 469us) service_pool.cc:225] Handling call
0623 01:59:17.747914 (+157448us) ts_tablet_manager.cc:1145] Deleting tablet 92909e3375c549b5baa0595e1750fa1f
0623 01:59:17.769831 (+ 21917us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 01:59:18.322540 (+552709us) tablet_metadata.cc:641] Metadata flushed
0623 01:59:18.649071 (+326531us) tablet_metadata.cc:641] Metadata flushed
0623 01:59:18.691483 (+ 42412us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78261,"tablet-delete.queue_time_us":136446,"thread_start_us":47071,"threads_started":1}
W20250623 01:59:18.861269 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 113299us
I20250623 01:59:18.881403 13694 catalog_manager.cc:4928] TS ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): tablet 92909e3375c549b5baa0595e1750fa1f (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
W20250623 01:59:18.929670 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 81838.6us
W20250623 01:59:18.953572 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 81551.2us
I20250623 01:59:18.989034 16164 raft_consensus.cc:991] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:18.989524 16164 raft_consensus.cc:1079] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:18.991832 13881 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:18.992347 13881 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:18.992709 13881 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:19.017990 13881 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:19.035840 13881 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:19.040920 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:19.041630 13806 raft_consensus.cc:3053] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:19.041924 13806 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
W20250623 01:59:19.061813 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 192399us
I20250623 01:59:19.062372 13806 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:19.063625 13806 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:19.083556 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:19.084206 13953 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:19.103838 16176 raft_consensus.cc:991] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:19.104507 16261 raft_consensus.cc:1079] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:19.108738 13813 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48570
I20250623 01:59:19.109402 13813 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:19.109836 13813 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:19.112335 13953 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:19.122378 13813 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:19.131299 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:19.131856 13883 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:19.133976 13813 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:19.137769 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:19.138348 13957 raft_consensus.cc:3053] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:19.138605 13957 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:19.139322 13957 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:19.140687 13957 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:19.124545 13854 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:19.141991 16089 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:19.142454 16089 raft_consensus.cc:695] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:19.143250 16089 consensus_queue.cc:237] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:19.145345 13803 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ac15d89950f4485a84912743c361f1d with delete_type TABLET_DATA_TOMBSTONED (TS 85a58a834b6b4b0dbe197b3dad87e382 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48562
I20250623 01:59:19.147043 13883 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:19.149542 13957 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:19.151273 13776 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:19.152992 16222 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:19.153496 16222 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:19.154309 16222 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:19.186022 16266 tablet_replica.cc:331] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:19.186589 13806 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:19.187350 16266 raft_consensus.cc:2241] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:19.188226 16266 raft_consensus.cc:2270] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:19.193893 16266 ts_tablet_manager.cc:1905] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:19.210829 13709 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 7 to 8, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 8 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:19.229591 13710 catalog_manager.cc:5582] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 6, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:19.235417 13806 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:19.239195 16266 ts_tablet_manager.cc:1918] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.6
I20250623 01:59:19.256942 16266 log.cc:1199] T 6ac15d89950f4485a84912743c361f1d P 85a58a834b6b4b0dbe197b3dad87e382: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-0-root/wals/6ac15d89950f4485a84912743c361f1d
I20250623 01:59:19.259622 14508 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250623 01:59:19.260340 13883 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250623 01:59:19.261224 16222 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250623 01:59:19.261744 13695 catalog_manager.cc:4928] TS 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): tablet 6ac15d89950f4485a84912743c361f1d (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:19.275714 16212 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250623 01:59:19.285524 16091 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Committing config change with OpId 8.9: config changed from index 8 to 9, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:19.301301 13883 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 8 to 9, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:19.319737 13695 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1b3574beac134b2eabbd65a4f54b3ab3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:59:19.325513 13710 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index 8 to 9, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New cstate: current_term: 8 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:19.315254 14508 raft_consensus.cc:2953] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 8 to 9, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:19.369055 13951 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b3574beac134b2eabbd65a4f54b3ab3 with delete_type TABLET_DATA_TOMBSTONED (TS c243715fe12445019c33bbeafbaea635 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37408
I20250623 01:59:19.386102 16269 tablet_replica.cc:331] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: stopping tablet replica
I20250623 01:59:19.403077 16269 raft_consensus.cc:2241] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:19.403898 16269 raft_consensus.cc:2270] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:19.414861 16269 ts_tablet_manager.cc:1905] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:19.438834 16269 ts_tablet_manager.cc:1918] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 8.8
I20250623 01:59:19.439373 16269 log.cc:1199] T 1b3574beac134b2eabbd65a4f54b3ab3 P c243715fe12445019c33bbeafbaea635: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/wals/1b3574beac134b2eabbd65a4f54b3ab3
I20250623 01:59:19.445456 13697 catalog_manager.cc:4928] TS c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): tablet 1b3574beac134b2eabbd65a4f54b3ab3 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:19.560600 13811 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:19.682861 16226 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:19.800465 13953 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:19.807891 13958 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:19.809787 16243 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250623 01:59:19.985689 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 169238us
W20250623 01:59:20.000602 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 46486us
I20250623 01:59:20.010478 16263 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:20.016839 13879 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20250623 01:59:20.107647 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 70059.4us
W20250623 01:59:20.122721 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 103502us
I20250623 01:59:20.124294 16245 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250623 01:59:20.131991 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 108695us
W20250623 01:59:20.251240 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 45631.9us
W20250623 01:59:20.251912 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 208892us
W20250623 01:59:20.260206 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 227627us
I20250623 01:59:20.273437 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:20.452203 13806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:20.452819 13806 raft_consensus.cc:604] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:20.455401 13885 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:20.455931 13885 raft_consensus.cc:604] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:20.458525 13806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:20.459024 13806 raft_consensus.cc:604] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:20.460186 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:20.462015 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:20.462765 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:20.537753 16091 raft_consensus.cc:991] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:20.538265 16091 raft_consensus.cc:1079] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:20.541754 13883 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:20.542326 13883 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:20.542959 13883 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:20.555692 13883 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.564738 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:20.565333 13806 raft_consensus.cc:3053] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:20.565680 13806 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:20.566342 13806 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.567319 13806 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:20.568001 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:20.568543 13956 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:20.576622 13742 debug-util.cc:398] Leaking SignalData structure 0x7b08002e3420 after lost signal to thread 12639
W20250623 01:59:20.577728 13742 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8f00 after lost signal to thread 13745
W20250623 01:59:20.578888 13742 debug-util.cc:398] Leaking SignalData structure 0x7b0800036f60 after lost signal to thread 13837
W20250623 01:59:20.579944 13742 debug-util.cc:398] Leaking SignalData structure 0x7b0800036700 after lost signal to thread 13912
W20250623 01:59:20.580955 13742 debug-util.cc:398] Leaking SignalData structure 0x7b0800315d60 after lost signal to thread 13986
I20250623 01:59:20.582054 13806 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:20.583292 13852 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:20.584359 13742 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba4e0 after lost signal to thread 14540
I20250623 01:59:20.585966 13883 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:20.591007 13956 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:20.591765 16226 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:20.592270 16226 raft_consensus.cc:695] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:20.593041 16226 consensus_queue.cc:237] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.621973 13709 catalog_manager.cc:5582] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 6, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:20.626048 13955 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:20.645589 13883 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20250623 01:59:20.646147 13927 consensus_peers.cc:487] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f023b481774a4d06bf5c5c62a3df3012. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:20.647248 16071 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:20.649724 13806 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250623 01:59:20.653090 16286 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:20.657721 16282 raft_consensus.cc:991] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:20.658320 16282 raft_consensus.cc:1079] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:20.660020 13806 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:20.660647 13806 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:20.660924 13806 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:20.668612 13806 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.673803 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:20.674153 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:20.674466 13885 raft_consensus.cc:3053] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:59:20.674377 13806 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:20.674916 13956 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:20.674976 13885 raft_consensus.cc:738] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:20.676136 13885 consensus_queue.cc:260] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.680639 13885 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:59:20.686678 13956 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:20.688714 13779 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:20.689604 16091 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:59:20.690062 16091 raft_consensus.cc:695] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:20.695690 16091 consensus_queue.cc:237] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.723065 13885 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:20.745558 13883 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:20.744709 16231 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:20.782666 13811 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:20.808058 13710 catalog_manager.cc:5582] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 1 to 2, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 2 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:20.814924 16279 raft_consensus.cc:991] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:20.815529 16279 raft_consensus.cc:1079] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:20.825995 13955 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37424
I20250623 01:59:20.826546 13955 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:20.826875 13955 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:20.829340 13959 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:20.831889 13708 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index -1 to 4, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 3 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:20.854065 13955 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.864195 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:20.865643 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:20.866151 13812 raft_consensus.cc:3053] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:20.866449 13812 raft_consensus.cc:738] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:20.869441 13955 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:20.864792 13883 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:20.877807 13812 consensus_queue.cc:260] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.879016 13883 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:20.879144 13812 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:20.882462 13885 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:20.883981 16176 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250623 01:59:20.885807 13927 consensus_peers.cc:487] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15e1fc4335c84e18ba2e5c6ea55fed7a. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:20.894246 13812 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:20.895395 13925 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:20.898357 16176 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:20.898893 16176 raft_consensus.cc:695] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:20.899782 16176 consensus_queue.cc:237] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:20.904996 13811 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:20.906456 16286 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:20.925068 16286 raft_consensus.cc:2953] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:20.930851 13879 raft_consensus.cc:2953] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:20.954227 13811 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:20.954180 13710 catalog_manager.cc:5582] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index -1 to 6, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 6 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:20.974740 13959 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20250623 01:59:20.975401 13778 consensus_peers.cc:487] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2928af9e5df4f9eacf59f499ab96dc1. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:20.976454 16277 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:20.976780 13879 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:20.984985 16091 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:59:20.991606 13709 catalog_manager.cc:5582] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 3, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 3 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:21.014467 13812 raft_consensus.cc:2953] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.031518 16279 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.039065 13885 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.063436 13710 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index -1 to 6, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.071502 13956 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:21.072057 13812 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:21.073239 16246 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:21.075366 16263 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:21.198931 13959 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.242909 16316 ts_tablet_manager.cc:927] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:21.278893 13956 raft_consensus.cc:1273] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:59:21.284302 16091 consensus_queue.cc:1035] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:59:21.375342 13879 raft_consensus.cc:1273] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:59:21.376487 13806 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:21.390986 13879 raft_consensus.cc:1273] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:21.392724 16107 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:21.410006 16277 consensus_queue.cc:1035] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:59:21.425424 16316 tablet_copy_client.cc:323] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.3:33369
I20250623 01:59:21.446568 13956 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:21.446914 13885 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:21.449913 16323 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:21.451557 16318 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:21.455030 13972 tablet_copy_service.cc:140] P c243715fe12445019c33bbeafbaea635: Received BeginTabletCopySession request for tablet f023b481774a4d06bf5c5c62a3df3012 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:34194)
I20250623 01:59:21.455930 13972 tablet_copy_service.cc:161] P c243715fe12445019c33bbeafbaea635: Beginning new tablet copy session on tablet f023b481774a4d06bf5c5c62a3df3012 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:34194: session id = 7d22741cfb9b408684cb82c58241b2f5-f023b481774a4d06bf5c5c62a3df3012
W20250623 01:59:21.455947 13778 consensus_peers.cc:487] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 -> Peer 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393): Couldn't send request to peer 7d22741cfb9b408684cb82c58241b2f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 318e5309795f47f3868e0787ddfdd146. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:21.472687 13972 tablet_copy_source_session.cc:215] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:59:21.495662 16316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f023b481774a4d06bf5c5c62a3df3012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:21.507078 13806 raft_consensus.cc:1273] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250623 01:59:21.509436 16231 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:21.557364 16323 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.560019 13885 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.563694 16327 ts_tablet_manager.cc:927] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:21.579463 16331 ts_tablet_manager.cc:927] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:59:21.582253 13709 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 reported cstate change: config changed from index -1 to 7, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New cstate: current_term: 8 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.592794 13956 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } }
I20250623 01:59:21.601148 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:21.607746 16316 tablet_copy_client.cc:806] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:59:21.608387 16316 tablet_copy_client.cc:670] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:59:21.630963 16316 tablet_copy_client.cc:538] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:59:21.633780 16327 tablet_copy_client.cc:323] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.3:33369
I20250623 01:59:21.646345 13972 tablet_copy_service.cc:140] P c243715fe12445019c33bbeafbaea635: Received BeginTabletCopySession request for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:34194)
I20250623 01:59:21.646925 13972 tablet_copy_service.cc:161] P c243715fe12445019c33bbeafbaea635: Beginning new tablet copy session on tablet 15e1fc4335c84e18ba2e5c6ea55fed7a from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:34194: session id = 7d22741cfb9b408684cb82c58241b2f5-15e1fc4335c84e18ba2e5c6ea55fed7a
I20250623 01:59:21.655431 13972 tablet_copy_source_session.cc:215] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:59:21.640756 16331 tablet_copy_client.cc:323] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.1:34781
I20250623 01:59:21.658979 16316 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:59:21.665279 13823 tablet_copy_service.cc:140] P 85a58a834b6b4b0dbe197b3dad87e382: Received BeginTabletCopySession request for tablet e2928af9e5df4f9eacf59f499ab96dc1 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:41526)
I20250623 01:59:21.666066 13823 tablet_copy_service.cc:161] P 85a58a834b6b4b0dbe197b3dad87e382: Beginning new tablet copy session on tablet e2928af9e5df4f9eacf59f499ab96dc1 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:41526: session id = 7d22741cfb9b408684cb82c58241b2f5-e2928af9e5df4f9eacf59f499ab96dc1
I20250623 01:59:21.666178 16327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:21.686182 13823 tablet_copy_source_session.cc:215] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:59:21.691350 16331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2928af9e5df4f9eacf59f499ab96dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:21.721024 16316 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:21.724042 16316 tablet_bootstrap.cc:492] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:59:21.724874 16316 ts_tablet_manager.cc:1397] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.066s user 0.028s sys 0.000s
I20250623 01:59:21.727882 16316 raft_consensus.cc:357] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:21.728967 16316 raft_consensus.cc:738] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:59:21.729806 16316 consensus_queue.cc:260] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:21.736860 16316 ts_tablet_manager.cc:1428] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250623 01:59:21.742959 13972 tablet_copy_service.cc:342] P c243715fe12445019c33bbeafbaea635: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-f023b481774a4d06bf5c5c62a3df3012 received from {username='slave'} at 127.0.0.1:34194
I20250623 01:59:21.743419 13972 tablet_copy_service.cc:434] P c243715fe12445019c33bbeafbaea635: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-f023b481774a4d06bf5c5c62a3df3012 on tablet f023b481774a4d06bf5c5c62a3df3012 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:21.751209 16331 tablet_copy_client.cc:806] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:59:21.751349 13806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:21.751976 13806 raft_consensus.cc:604] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:21.756476 13959 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:21.756985 13959 raft_consensus.cc:604] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:21.751987 16331 tablet_copy_client.cc:670] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:59:21.759711 16327 tablet_copy_client.cc:806] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:59:21.760313 16327 tablet_copy_client.cc:670] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:59:21.761909 16331 tablet_copy_client.cc:538] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:59:21.763960 16327 tablet_copy_client.cc:538] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:59:21.777061 16331 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:59:21.780946 13885 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:21.781620 13885 raft_consensus.cc:604] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:21.785012 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:21.788177 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:21.790544 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:21.822088 16327 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:59:21.826198 16331 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:21.848500 16331 tablet_bootstrap.cc:492] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:59:21.851214 16331 ts_tablet_manager.cc:1397] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.074s user 0.031s sys 0.005s
I20250623 01:59:21.867625 16331 raft_consensus.cc:357] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:21.868588 16331 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:59:21.869284 16331 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:21.873356 16331 ts_tablet_manager.cc:1428] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250623 01:59:21.890339 13823 tablet_copy_service.cc:342] P 85a58a834b6b4b0dbe197b3dad87e382: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-e2928af9e5df4f9eacf59f499ab96dc1 received from {username='slave'} at 127.0.0.1:41526
I20250623 01:59:21.890763 13823 tablet_copy_service.cc:434] P 85a58a834b6b4b0dbe197b3dad87e382: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-e2928af9e5df4f9eacf59f499ab96dc1 on tablet e2928af9e5df4f9eacf59f499ab96dc1 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:21.904655 16091 raft_consensus.cc:991] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:21.905154 16091 raft_consensus.cc:1079] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:21.923509 16327 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:21.924541 16327 tablet_bootstrap.cc:492] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:59:21.925292 16327 ts_tablet_manager.cc:1397] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.103s user 0.014s sys 0.018s
I20250623 01:59:21.928130 16327 raft_consensus.cc:357] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:21.929060 16327 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:59:21.929769 16327 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:21.932503 16327 ts_tablet_manager.cc:1428] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20250623 01:59:21.937376 13885 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:21.938061 13885 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:21.938515 13885 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:21.946622 13972 tablet_copy_service.cc:342] P c243715fe12445019c33bbeafbaea635: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-15e1fc4335c84e18ba2e5c6ea55fed7a received from {username='slave'} at 127.0.0.1:34194
I20250623 01:59:21.947093 13972 tablet_copy_service.cc:434] P c243715fe12445019c33bbeafbaea635: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-15e1fc4335c84e18ba2e5c6ea55fed7a on tablet 15e1fc4335c84e18ba2e5c6ea55fed7a with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:21.950860 16327 ts_tablet_manager.cc:927] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: Initiating tablet copy from peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:59:21.964560 16327 tablet_copy_client.cc:323] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.87.1:34781
I20250623 01:59:21.966063 13823 tablet_copy_service.cc:140] P 85a58a834b6b4b0dbe197b3dad87e382: Received BeginTabletCopySession request for tablet 318e5309795f47f3868e0787ddfdd146 from peer 7d22741cfb9b408684cb82c58241b2f5 ({username='slave'} at 127.0.0.1:41526)
I20250623 01:59:21.966555 13823 tablet_copy_service.cc:161] P 85a58a834b6b4b0dbe197b3dad87e382: Beginning new tablet copy session on tablet 318e5309795f47f3868e0787ddfdd146 from peer 7d22741cfb9b408684cb82c58241b2f5 at {username='slave'} at 127.0.0.1:41526: session id = 7d22741cfb9b408684cb82c58241b2f5-318e5309795f47f3868e0787ddfdd146
I20250623 01:59:21.976778 13823 tablet_copy_source_session.cc:215] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:59:21.979817 16327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 318e5309795f47f3868e0787ddfdd146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:21.990916 14508 raft_consensus.cc:1215] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Deduplicated request from leader. Original: 6.5->[6.6-6.6] Dedup: 6.6->[]
I20250623 01:59:22.014091 16327 tablet_copy_client.cc:806] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 0 data blocks...
I20250623 01:59:22.014508 16246 raft_consensus.cc:991] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:22.014854 16327 tablet_copy_client.cc:670] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Starting download of 1 WAL segments...
I20250623 01:59:22.015077 16246 raft_consensus.cc:1079] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:22.017568 13955 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:22.018358 13955 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:22.018776 13955 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:22.021189 16327 tablet_copy_client.cc:538] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:59:22.042106 13955 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.058243 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:22.057293 13955 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:22.058921 13888 raft_consensus.cc:3053] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:22.059304 13888 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:22.063802 14508 raft_consensus.cc:1215] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Deduplicated request from leader. Original: 6.5->[6.6-6.6] Dedup: 6.6->[]
I20250623 01:59:22.067189 13888 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.068432 13888 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:22.079828 13888 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:22.083139 13925 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:22.084091 16231 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:22.084578 16231 raft_consensus.cc:695] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:22.085361 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:22.085989 13806 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:22.085417 16231 consensus_queue.cc:237] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.092590 13806 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:22.094429 13885 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.097003 13885 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:22.098654 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:22.100528 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:22.102607 13806 raft_consensus.cc:3053] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:22.102772 13955 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:22.102980 13806 raft_consensus.cc:738] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:22.103549 13806 consensus_queue.cc:260] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.104288 13806 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:22.109738 16231 raft_consensus.cc:991] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:22.110010 13955 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:22.110215 16231 raft_consensus.cc:1079] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:22.115046 13854 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:22.116087 16280 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:22.116600 16280 raft_consensus.cc:695] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:22.117417 16280 consensus_queue.cc:237] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.120393 13811 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48570
I20250623 01:59:22.120952 13811 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:22.121273 13811 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:22.123689 16327 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap starting.
I20250623 01:59:22.100394 13710 catalog_manager.cc:5582] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:22.128065 13811 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.146448 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:22.148013 13885 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:22.155390 13811 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:22.166512 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:22.167098 13959 raft_consensus.cc:3053] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:22.167402 13959 raft_consensus.cc:738] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:22.168210 13959 consensus_queue.cc:260] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.154579 13885 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:22.170759 14508 raft_consensus.cc:1215] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20250623 01:59:22.175698 13959 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:22.182106 13776 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:22.183080 16323 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:22.183580 16323 raft_consensus.cc:695] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:22.184389 16323 consensus_queue.cc:237] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:22.206733 13959 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:22.231738 13806 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:22.242050 13708 catalog_manager.cc:5582] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 7, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:22.243963 13710 catalog_manager.cc:5582] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 3 to 4, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 4 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:22.237243 16327 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=1 applied=7 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:22.262888 16327 tablet_bootstrap.cc:492] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: Bootstrap complete.
I20250623 01:59:22.263640 16327 ts_tablet_manager.cc:1397] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: Time spent bootstrapping tablet: real 0.140s user 0.027s sys 0.008s
I20250623 01:59:22.276605 16327 raft_consensus.cc:357] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:22.293046 16327 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Initialized, Role: LEARNER
I20250623 01:59:22.293834 16327 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:22.298839 16327 ts_tablet_manager.cc:1428] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: Time spent starting tablet: real 0.035s user 0.003s sys 0.004s
I20250623 01:59:22.301092 13823 tablet_copy_service.cc:342] P 85a58a834b6b4b0dbe197b3dad87e382: Request end of tablet copy session 7d22741cfb9b408684cb82c58241b2f5-318e5309795f47f3868e0787ddfdd146 received from {username='slave'} at 127.0.0.1:41526
I20250623 01:59:22.301558 13823 tablet_copy_service.cc:434] P 85a58a834b6b4b0dbe197b3dad87e382: ending tablet copy session 7d22741cfb9b408684cb82c58241b2f5-318e5309795f47f3868e0787ddfdd146 on tablet 318e5309795f47f3868e0787ddfdd146 with peer 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:22.446591 16277 raft_consensus.cc:1062] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:59:22.472980 16277 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:22.527110 16231 raft_consensus.cc:1062] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:59:22.530551 16231 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:22.537200 14508 raft_consensus.cc:1215] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEARNER]: Deduplicated request from leader. Original: 8.6->[8.7-8.7] Dedup: 8.7->[]
I20250623 01:59:22.548395 13806 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:22.556833 13812 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:22.563529 16176 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:22.566373 16341 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:22.612839 14508 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:22.615520 13888 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:22.619874 14509 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 LEARNER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:22.623255 16176 raft_consensus.cc:1062] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:59:22.623991 16318 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:22.626178 16107 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:22.629792 16323 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:22.639788 13959 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:22.646633 16176 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:22.655021 13812 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:22.656879 16323 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:22.661674 13885 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:22.663455 13883 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:22.666561 13953 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:22.690590 16341 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:22.690009 16231 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20250623 01:59:22.695792 16286 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:22.695966 16352 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:22.701592 13953 raft_consensus.cc:1273] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:22.717736 16318 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20250623 01:59:22.817345 14509 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEARNER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250623 01:59:22.827260 16176 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20250623 01:59:22.909777 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:22.934079 13883 raft_consensus.cc:1273] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:22.951493 13885 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250623 01:59:24.028450 16231 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.068s user 0.004s sys 0.014s
W20250623 01:59:24.143158 16231 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.185s user 0.005s sys 0.014s
I20250623 01:59:24.651613 16107 raft_consensus.cc:2953] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:24.691195 16312 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.003s sys 0.020s
W20250623 01:59:24.691787 16312 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.768s user 0.003s sys 0.021s
W20250623 01:59:24.760967 14478 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.833s user 0.030s sys 0.087s
W20250623 01:59:24.761390 14478 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.833s user 0.031s sys 0.087s
W20250623 01:59:24.825784 14479 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.897s user 0.005s sys 0.005s
W20250623 01:59:24.826476 14479 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.898s user 0.005s sys 0.005s
W20250623 01:59:24.827152 14481 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.869s user 0.000s sys 0.017s
W20250623 01:59:24.827539 14481 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.899s user 0.000s sys 0.017s
I20250623 01:59:24.830181 16387 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:24.830600 16390 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:24.836947 16390 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.835844 16387 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.839954 16387 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.840143 16390 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.875432 16391 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:24.877997 16391 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.882140 16390 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:24.882524 16391 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:24.883278 16391 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:24.883241 16390 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.884137 16391 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.887545 16390 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.887180 16391 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.888409 16392 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:24.889623 16392 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.894389 16392 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.897001 16391 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:24.898029 16391 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.905099 16391 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.906468 16394 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:24.910650 16394 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.919606 16394 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:24.919694 16385 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:24.920708 16385 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.923508 16385 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:24.930130 16385 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:24.931062 16385 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:24.933888 16385 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.935506 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:24.939378 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:24.941762 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:24.943729 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:24.945327 16376 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:24.945472 16378 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:24.946336 16376 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:24.946498 16378 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:24.949862 16378 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.950139 16376 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.944435 16385 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:24.950863 16376 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:24.951269 16385 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:24.951833 16376 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:24.953956 16385 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.954828 16376 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.953019 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:24.955972 16378 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:24.956121 13808 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:24.951803 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:24.957757 16376 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:24.958050 13804 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:24.958667 16376 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:24.956845 16378 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:24.964221 13925 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 2.030s user 0.021s sys 0.032s
W20250623 01:59:24.964900 13925 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 2.031s user 0.021s sys 0.032s
I20250623 01:59:24.966747 16378 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.959221 16385 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:24.960018 16371 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:24.967957 16371 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:24.970759 16371 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.975412 16286 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:24.961138 16379 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:24.978387 16379 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:24.986358 16385 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:24.988969 14479 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:59:24.995625 13956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 1908) took 2075 ms. Trace:
I20250623 01:59:24.996361 16394 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:25.010695 16394 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.011230 16394 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:25.013578 13956 rpcz_store.cc:276] 0623 01:59:22.920107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:22.920286 (+ 179us) service_pool.cc:225] Handling call
0623 01:59:22.920686 (+ 400us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:22.921120 (+ 434us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:22.921156 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:22.923848 (+ 2692us) log.cc:844] Serialized 26 byte log entry
0623 01:59:22.970820 (+ 46972us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:59:22.970863 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:22.970934 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:24.995123 (+2024189us) raft_consensus.cc:1683] finished
0623 01:59:24.995162 (+ 39us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:24.995385 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":219,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9530,"thread_start_us":46621,"threads_started":1,"wal-append.queue_time_us":46941,"wal-append.run_cpu_time_us":1918,"wal-append.run_wall_time_us":1904253}
I20250623 01:59:24.988165 16376 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:24.981117 16379 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.023980 16286 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.031994 16286 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.002180 16385 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.035142 16396 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.033061 16394 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.036101 16396 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.038846 16396 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.039336 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:25.039530 16396 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.040033 13804 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:25.040694 16396 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.041244 16286 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.042001 16286 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.042887 16396 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.043500 16396 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.044108 16286 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.044170 16396 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.045368 16107 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.045894 16107 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.046517 16396 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.047080 16396 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.047799 16107 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.047652 16396 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:25.048103 16381 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.048368 16107 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.048869 16381 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.049888 16231 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.049978 16396 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.050669 16396 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.051002 16381 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.050653 16231 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.049080 16107 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.051450 16396 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.053005 16231 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.053020 16107 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.053541 16396 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.050990 13804 raft_consensus.cc:2466] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:25.055656 14479 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:59:25.056059 16107 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:25.053364 13776 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 2.096s user 0.003s sys 0.030s
W20250623 01:59:25.056803 13776 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 2.100s user 0.004s sys 0.030s
I20250623 01:59:25.056766 16107 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.057329 16391 raft_consensus.cc:2802] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:25.057956 16391 raft_consensus.cc:695] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:25.059005 16107 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.059187 16286 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:25.057329 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 2.10008e+06us
I20250623 01:59:25.062587 16369 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:25.065133 16231 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.065960 16231 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.067909 16231 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.059568 16107 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.059157 16391 consensus_queue.cc:237] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.060010 16286 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.073387 16107 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:25.071648 16318 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:25.086114 16107 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.100742 16369 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.101483 16369 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.103941 16369 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.105038 16376 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.105796 16376 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.108179 16376 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.108860 16376 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.109638 16376 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:25.111711 16376 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.112865 16323 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.113458 16323 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:25.115370 16323 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.074975 16286 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.075518 16385 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.069795 16394 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.119764 16397 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.120509 16397 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.122694 16397 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.124783 16376 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.125823 16376 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.117540 16176 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:25.128924 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.130807 16369 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.131006 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.131492 16369 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.132934 16277 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.126047 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.126365 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.118067 16385 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.126798 14509 raft_consensus.cc:2953] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:25.133656 16277 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.137140 16277 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.135099 14506 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:25.133615 16369 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.138185 16376 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.138943 13927 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:25.140067 13779 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 2.203s user 0.016s sys 0.020s
W20250623 01:59:25.140444 13779 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 2.203s user 0.016s sys 0.020s
W20250623 01:59:25.145563 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 2.20821e+06us
I20250623 01:59:25.146350 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.147033 13813 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:59:25.153443 16396 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:25.153939 16396 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.154341 16396 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:25.157588 16286 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.158480 16231 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.158337 16286 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:25.160035 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 2.18307e+06us
I20250623 01:59:25.160440 16107 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.161054 16107 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.162166 16381 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.162799 16381 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.170537 16397 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.171234 16397 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.173792 16397 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.181486 16379 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.182139 16379 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.184231 16379 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.188359 16378 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.191614 16376 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.192356 16376 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.188980 16378 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.197599 16378 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.174854 16385 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.176501 16381 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.156333 16398 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.199316 16398 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.179172 13804 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250623 01:59:25.203302 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.203974 14510 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:25.204685 16376 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.206596 16398 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.206919 13927 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:25.223910 16385 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.211946 16398 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:25.225804 16385 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.225800 16398 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.227965 16398 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:25.234164 16385 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.243011 16401 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.243878 16401 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.246265 16401 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.237687 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.247004 16176 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.247992 16401 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.248342 16176 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.248741 16397 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.249491 16397 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.250623 16176 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.247016 16398 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.241847 16381 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.253336 16385 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.253664 16381 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.254113 16398 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.253970 16385 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.255771 16381 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.256201 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:25.256800 14511 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:25.261613 16397 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.264891 14511 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:25.261716 16385 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.235591 16396 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.269636 16318 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.270234 16369 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.277415 16369 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.279729 16369 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.277367 16318 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.283625 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 252) took 2333 ms. Trace:
I20250623 01:59:25.284319 13708 rpcz_store.cc:276] 0623 01:59:22.949803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:22.949942 (+ 139us) service_pool.cc:225] Handling call
0623 01:59:25.283185 (+2333243us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20250623 01:59:25.285535 16318 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.286962 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.287904 16396 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.288093 13927 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:25.249451 16401 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.299511 16382 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.300127 16382 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.300855 16401 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.298702 16397 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.304711 16397 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.306787 16397 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.298851 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:25.307960 14507 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:25.297026 16396 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:25.303706 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.309028 16396 raft_consensus.cc:695] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:25.310247 16396 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.310166 14514 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:25.303874 16382 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.317592 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.345217 16406 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.345939 16406 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.346714 16401 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.346946 16407 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.347417 16401 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.347556 16407 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.348213 16406 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.349673 16401 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.350069 16382 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.350217 16401 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.350665 16382 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.350773 16401 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.353928 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.358860 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.368287 16107 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.373425 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.374053 14509 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:25.368144 16382 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.376109 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.376965 13813 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:25.377993 13927 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:25.369694 16406 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.379987 16406 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.380808 16401 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.382022 16406 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.382076 16397 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:25.382905 16397 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.383479 14507 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:25.384441 14515 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:25.385927 16397 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.360563 16407 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.387923 13927 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:25.388886 16407 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:25.389281 16407 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.389442 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.389762 16407 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:25.390146 13812 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:25.393738 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.395457 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.394469 13813 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:25.398519 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.399207 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.398593 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.399554 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.398612 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.404361 16318 raft_consensus.cc:1062] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: attempting to promote NON_VOTER 7d22741cfb9b408684cb82c58241b2f5 to VOTER
I20250623 01:59:25.401898 13806 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:25.404593 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.405818 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.405902 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.406443 13806 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:25.407533 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.408278 13811 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:25.407696 13809 raft_consensus.cc:2953] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:25.409126 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.409675 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.409771 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.410202 13804 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:25.410279 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.410434 13805 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:25.411360 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.411525 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.411617 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.411931 13811 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:25.412189 13808 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:25.413911 16398 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:25.414953 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.415530 13959 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:25.415652 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.416056 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.416759 13960 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:25.416910 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.397507 16397 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:25.419005 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.419611 13956 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:25.421550 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.423486 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.423815 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.424137 13961 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:25.424336 13954 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:25.423503 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.424996 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.425539 13812 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:25.406859 16318 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.430444 16361 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.431058 16361 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.431478 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.434613 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.433923 16361 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.425976 16396 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.437765 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.439880 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.445612 13961 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:25.446233 16312 consensus_queue.cc:1046] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250623 01:59:25.447393 16312 raft_consensus.cc:3053] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:25.447722 16312 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:25.423076 16397 raft_consensus.cc:695] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:25.449685 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.448625 16312 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.454747 13962 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:25.457196 13962 raft_consensus.cc:604] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:25.455451 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.458489 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.460479 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.460855 16312 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:25.461441 16396 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:25.467337 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 2.49649e+06us
I20250623 01:59:25.470439 16396 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.472229 13709 catalog_manager.cc:5582] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 6 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 7 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:25.477945 13709 catalog_manager.cc:5582] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:25.482223 16397 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.159209 16231 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.490139 16231 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.504225 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.506261 14481 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:25.507110 16280 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.507773 16280 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.508210 16368 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.508354 16404 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.515110 14481 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:25.508848 16368 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.519884 16246 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.520546 16246 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.521473 16368 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.521652 16405 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.522203 16368 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.522547 16246 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.522257 16405 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.523764 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.524529 16405 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.525174 16384 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:25.529695 16384 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:25.533118 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:25.534493 16280 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.537449 16341 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.509496 16286 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:25.546702 16341 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.548856 16341 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.548884 16405 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.550082 16405 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.552075 16405 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.513875 16404 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.557431 16404 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.558182 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.558802 13956 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:25.546981 16361 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.561336 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.561118 16361 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.563577 16361 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.548949 16364 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.564776 16364 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.565966 13954 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:25.567057 16364 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.566869 13708 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:25.568593 13806 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250623 01:59:25.570158 16389 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250623 01:59:25.573117 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.566979 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.578989 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.523017 16368 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.584223 13811 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:25.584434 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.584810 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.583297 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.584782 13811 raft_consensus.cc:604] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:25.587217 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.587792 13956 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:25.594655 13812 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:25.594523 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:25.594496 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.589250 16403 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:25.605624 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 137211us
I20250623 01:59:25.604988 16361 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.612453 16361 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.602351 16407 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.613988 14506 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEARNER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:25.614289 13955 raft_consensus.cc:1238] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:25.614521 16361 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.615600 16371 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:25.618458 16364 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:25.618884 16364 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.619009 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.619448 16364 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:25.629459 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:25.634115 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.622958 16403 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:25.633358 16280 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.636811 16403 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.636752 16280 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.638669 16280 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.638021 16368 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.642649 16407 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.643610 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:25.651904 16368 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.652463 16368 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.656926 16403 consensus_queue.cc:1046] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:25.659123 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.659974 13961 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:25.664134 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.664944 16280 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:25.665628 16280 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.667014 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.667654 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.668090 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.672128 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.674021 16368 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.677831 16364 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.680661 16280 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:25.681352 16403 consensus_queue.cc:1230] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [LEADER]: Peer c243715fe12445019c33bbeafbaea635 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250623 01:59:25.682802 16403 raft_consensus.cc:3053] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:25.683106 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.683228 16403 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:25.684345 16403 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.694548 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.696295 16403 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:25.696619 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.698220 13883 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:25.698305 14507 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:25.693135 13885 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:25.725594 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.708423 13885 raft_consensus.cc:604] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:25.727337 16418 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.728102 16418 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.730120 16360 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.730739 16360 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.732784 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.738615 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.739346 13882 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:25.739827 16420 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.740592 16420 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:25.740684 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.740918 16402 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:25.757237 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:25.860316 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.903002 13959 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:25.814323 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:25.908548 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:25.878696 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.799052 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.910383 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:25.903913 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.911687 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.914506 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.915124 13888 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:25.915987 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.916566 13888 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:25.917420 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:25.912534 13885 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
I20250623 01:59:25.810278 16360 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.917892 16364 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:25.900919 16402 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:25.997100 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.035458 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:25.940121 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.977855 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.975128 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.973573 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.069470 13854 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:25.977186 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.930105 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:25.965575 13880 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 3.
I20250623 01:59:26.035141 16418 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:26.116591 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:26.293921 13881 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:25.939966 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.295034 13887 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
I20250623 01:59:25.742009 16405 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:25.930410 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.296113 16405 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.976657 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.718171 16368 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:59:26.188856 13983 debug-util.cc:398] Leaking SignalData structure 0x7b08001bd840 after lost signal to thread 12639
I20250623 01:59:26.297716 16368 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:25.977030 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.253681 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:26.299163 13983 debug-util.cc:398] Leaking SignalData structure 0x7b080026a5e0 after lost signal to thread 13745
I20250623 01:59:25.742130 16361 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:25.957876 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:26.299754 16368 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.234872 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:26.300804 13983 debug-util.cc:398] Leaking SignalData structure 0x7b08003392e0 after lost signal to thread 13837
I20250623 01:59:25.941088 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.301725 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:26.301982 13885 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:26.303038 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:25.741552 16246 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:25.984127 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:25.951301 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.270152 14478 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:26.308259 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.298146 16405 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.309443 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.304860 16364 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.314268 16364 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.315228 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.315728 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.316236 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.316746 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:59:26.319685 13983 debug-util.cc:398] Leaking SignalData structure 0x7b080030a160 after lost signal to thread 13912
W20250623 01:59:26.320744 13983 debug-util.cc:398] Leaking SignalData structure 0x7b08000483a0 after lost signal to thread 13986
I20250623 01:59:26.307742 13953 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:26.321887 13983 debug-util.cc:398] Leaking SignalData structure 0x7b08003aeb80 after lost signal to thread 14540
I20250623 01:59:26.326467 16368 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:26.326959 16368 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:26.327365 16368 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:26.329689 16404 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.330193 16404 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.316247 16364 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.331923 16404 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.332633 16404 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:26.326381 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 256983us
I20250623 01:59:26.333487 13955 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:26.333102 16404 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.335502 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.335974 16404 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.340389 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.341431 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.342581 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.343796 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.347194 13883 raft_consensus.cc:3053] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:26.350752 13883 raft_consensus.cc:738] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:26.351228 16405 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.351737 13883 consensus_queue.cc:260] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:26.352669 16404 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.352903 13883 raft_consensus.cc:3058] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:26.350119 16280 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:26.349516 16364 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.353150 16404 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.349902 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.353771 16280 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:26.348567 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.356606 16364 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.357679 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.358793 16404 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.359504 16364 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.359611 16420 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.357131 16280 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:26.353379 16360 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.360535 16360 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.361598 13882 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:26.353017 16368 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } }
I20250623 01:59:26.351799 16405 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.363617 13888 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:26.366137 16405 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.367558 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.370355 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.370934 16402 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:26.371642 16246 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:26.372754 13884 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:26.357625 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.375438 16246 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:26.375814 16418 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:26.376550 13886 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:26.383034 13883 raft_consensus.cc:1273] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250623 01:59:26.383924 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.392076 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.391345 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.394424 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.394809 13880 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:26.397198 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.397147 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:26.402258 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.403122 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.404067 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.375844 16368 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.407086 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.405709 13954 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:26.405359 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.413057 16402 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:26.413743 16402 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.299754 16361 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.421128 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.422215 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.422792 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.423138 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.423859 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.431340 16361 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.424247 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.425004 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.437722 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.443516 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.446496 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:26.453843 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 524575us
I20250623 01:59:26.454854 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.463042 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.464315 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.448390 16360 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:26.464917 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 58928.9us
I20250623 01:59:26.404656 16418 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.482986 14478 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:26.458213 16402 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.482633 13883 raft_consensus.cc:2953] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:26.464519 16425 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:26.515354 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:26.515281 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:26.523193 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:26.534858 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:26.542001 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.542654 13886 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:26.549386 13926 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:26.534078 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:26.550575 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 107977us
I20250623 01:59:26.551419 13926 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.552392 13926 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.504952 16425 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:59:26.554713 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 85445.1us
I20250623 01:59:26.557377 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.555548 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.567826 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.553057 16386 raft_consensus.cc:2762] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20250623 01:59:26.574556 16412 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:26.575136 16412 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.575577 16412 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:26.576876 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.577652 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.578357 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.578722 16437 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:26.579178 16437 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.579546 16437 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:26.579849 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.570124 16286 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:26.582209 16418 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.583917 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:26.584843 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.585206 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.586000 13880 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
W20250623 01:59:26.588676 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:26.556536 16425 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:26.571740 16407 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:26.571877 16396 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:26.599167 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.599514 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:26.600827 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 43577.7us
I20250623 01:59:26.601171 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.603286 13926 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.598734 16407 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.604171 13926 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:26.598635 16396 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.605072 16396 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:26.604635 16286 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:26.609891 16286 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.610244 16286 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:26.612444 16407 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:26.613807 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.614373 13880 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:26.617969 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.618141 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.625942 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:26.626847 13925 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 1.509s (SENT)
I20250623 01:59:26.638353 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:26.639765 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 72018.9us
I20250623 01:59:26.640890 13779 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.641798 16371 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:26.642270 16371 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.642758 16371 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:26.650043 16371 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.652094 16371 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.653096 16371 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.653553 16371 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.652582 13779 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.656456 13779 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:26.655368 16371 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.657860 16371 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:26.658247 16371 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.658929 16371 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
W20250623 01:59:26.670696 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 187766us
I20250623 01:59:26.672501 14478 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:26.673200 13884 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:26.637609 16440 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:26.678879 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.677475 14478 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:26.676647 16388 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:26.690097 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.693902 14478 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:26.702093 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 59984.3us
I20250623 01:59:26.638094 16412 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:26.722019 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 95173.3us
I20250623 01:59:26.678000 16444 raft_consensus.cc:2747] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:26.690500 16440 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.734544 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.735898 13884 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 9: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:26.699759 16388 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.742507 16384 consensus_queue.cc:1035] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20250623 01:59:26.738858 16440 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
W20250623 01:59:26.767230 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 110836us
W20250623 01:59:26.771374 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:26.732825 16322 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:26.776863 14509 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:26.779906 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.780695 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.786924 13779 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:26.779078 16322 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:26.797662 16322 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:26.730913 16438 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:26.757193 16388 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:26.837908 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52432 (request call id 257) took 1339 ms. Trace:
I20250623 01:59:26.739084 16445 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:26.854301 14509 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250623 01:59:26.801899 16444 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:26.856393 16444 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:26.856846 16444 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:26.826167 16438 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:26.871245 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.854050 16445 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:26.868667 16424 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250623 01:59:26.894543 16449 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250623 01:59:26.858600 16389 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:26.896946 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.898700 13884 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:26.885807 16445 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:26.907517 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 120688us
W20250623 01:59:26.916224 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:26.880825 16438 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:26.934394 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.924901 16432 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:26.895431 16452 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:26.838554 13708 rpcz_store.cc:276] 0623 01:59:25.498118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:25.498352 (+ 234us) service_pool.cc:225] Handling call
0623 01:59:26.837761 (+1339409us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:25.951229 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:25.981877 (+ 30648us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:25.981939 (+ 62us) write_op.cc:435] Acquired schema lock
0623 01:59:25.981962 (+ 23us) tablet.cc:660] Decoding operations
0623 01:59:25.982360 (+ 398us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:25.982423 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:25.982462 (+ 39us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:59:25.982668 (+ 206us) tablet.cc:699] Row locks acquired
0623 01:59:25.982695 (+ 27us) write_op.cc:260] PREPARE: finished
0623 01:59:25.982850 (+ 155us) write_op.cc:270] Start()
0623 01:59:25.982986 (+ 136us) write_op.cc:276] Timestamp: P: 1750643965982798 usec, L: 0
0623 01:59:25.983010 (+ 24us) op_driver.cc:348] REPLICATION: starting
0623 01:59:25.983687 (+ 677us) log.cc:844] Serialized 591 byte log entry
0623 01:59:26.613566 (+629879us) op_driver.cc:464] REPLICATION: finished
0623 01:59:26.719961 (+106395us) write_op.cc:301] APPLY: starting
0623 01:59:26.730934 (+ 10973us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:26.733977 (+ 3043us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:26.737475 (+ 3498us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:26.737767 (+ 292us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:26.742043 (+ 4276us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:59:26.742106 (+ 63us) write_op.cc:312] APPLY: finished
0623 01:59:26.742397 (+ 291us) log.cc:844] Serialized 23 byte log entry
0623 01:59:26.742602 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:26.742675 (+ 73us) write_op.cc:454] Released schema lock
0623 01:59:26.742713 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:26.743072 (+ 359us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:26.743097 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":68090,"apply.run_cpu_time_us":2628,"apply.run_wall_time_us":52836,"num_ops":1,"prepare.queue_time_us":348308,"prepare.run_cpu_time_us":2920,"prepare.run_wall_time_us":42572,"replication_time_us":621325,"thread_start_us":15056,"threads_started":1}]]}
I20250623 01:59:26.907761 16457 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:59:26.966811 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 348907us
I20250623 01:59:26.961530 16412 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:26.983134 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.908726 16389 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:26.990584 13880 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:26.993580 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.973671 16452 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:26.997795 16452 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.998729 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 114) took 1653 ms. Trace:
I20250623 01:59:26.975564 16457 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:27.001830 16450 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20250623 01:59:27.002190 13710 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 7 to 8, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 8 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:26.999135 13709 rpcz_store.cc:276] 0623 01:59:25.345209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:25.345456 (+ 247us) service_pool.cc:225] Handling call
0623 01:59:26.998671 (+1653215us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:25.589128 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:25.618694 (+ 29566us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:25.618750 (+ 56us) write_op.cc:435] Acquired schema lock
0623 01:59:25.618841 (+ 91us) tablet.cc:660] Decoding operations
0623 01:59:25.619529 (+ 688us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:25.619606 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:25.619634 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0623 01:59:25.620077 (+ 443us) tablet.cc:699] Row locks acquired
0623 01:59:25.620118 (+ 41us) write_op.cc:260] PREPARE: finished
0623 01:59:25.620292 (+ 174us) write_op.cc:270] Start()
0623 01:59:25.620445 (+ 153us) write_op.cc:276] Timestamp: P: 1750643965620225 usec, L: 0
0623 01:59:25.620482 (+ 37us) op_driver.cc:348] REPLICATION: starting
0623 01:59:25.653522 (+ 33040us) log.cc:844] Serialized 968 byte log entry
0623 01:59:26.536418 (+882896us) op_driver.cc:464] REPLICATION: finished
0623 01:59:26.670292 (+133874us) write_op.cc:301] APPLY: starting
0623 01:59:26.674995 (+ 4703us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:26.698995 (+ 24000us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:26.744167 (+ 45172us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:26.832136 (+ 87969us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:26.832384 (+ 248us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 01:59:26.832450 (+ 66us) write_op.cc:312] APPLY: finished
0623 01:59:26.832672 (+ 222us) log.cc:844] Serialized 29 byte log entry
0623 01:59:26.832857 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:26.832964 (+ 107us) write_op.cc:454] Released schema lock
0623 01:59:26.833012 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:26.833406 (+ 394us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:26.833442 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":123641,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":133594,"apply.run_cpu_time_us":2565,"apply.run_wall_time_us":163135,"num_ops":2,"prepare.queue_time_us":103286,"prepare.run_cpu_time_us":17260,"prepare.run_wall_time_us":323935,"raft.queue_time_us":176450,"raft.run_cpu_time_us":16578,"raft.run_wall_time_us":91608,"replication_time_us":915738,"thread_start_us":193938,"threads_started":4,"wal-append.queue_time_us":714822}]]}
I20250623 01:59:27.015759 13710 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:27.028218 13957 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:27.031004 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:26.963696 16454 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20250623 01:59:27.060838 16389 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:26.983389 16407 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.040189 16440 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.043735 16371 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.045526 16396 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.047364 16457 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.050426 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.050843 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.051448 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.053655 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.054276 16437 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.054749 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.064654 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:27.074348 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.075601 16322 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:27.076635 16371 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.077459 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.078595 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.078891 16396 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:27.079542 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.080240 13879 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 9: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:27.081389 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.082314 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.084259 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.089607 14481 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:27.031787 13881 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
W20250623 01:59:27.069593 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 157870us
I20250623 01:59:27.015177 16286 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:27.096561 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.081050 16437 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:27.099620 13778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 174147us
I20250623 01:59:27.098807 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:27.101374 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:27.003898 16425 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:27.102741 14507 raft_consensus.cc:1238] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
W20250623 01:59:27.101970 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:27.105670 14506 raft_consensus.cc:1238] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:27.108489 16407 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:27.108791 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:27.109746 16459 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:27.110108 16459 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:27.110464 16459 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:27.110806 13887 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:27.111747 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.112704 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.113752 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.114739 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.114816 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.115729 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.115978 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.117251 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.118368 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.118688 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.120426 16437 consensus_queue.cc:1046] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:27.121590 16371 raft_consensus.cc:3053] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:27.121937 16371 raft_consensus.cc:738] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:27.108870 16440 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.122684 16371 consensus_queue.cc:260] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:27.124065 16371 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:27.126020 16445 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.126884 16286 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:27.129717 13779 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:27.070029 16402 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20250623 01:59:27.098759 16390 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:27.132992 16459 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.148944 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.155120 16444 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.156116 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:27.171583 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:27.147397 16440 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:27.201063 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.215272 13955 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:27.216269 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:59:27.246383 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:27.292003 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 123727us
W20250623 01:59:27.229451 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 260318us
I20250623 01:59:27.315330 13926 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:27.316097 16286 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:27.316496 16286 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:27.316879 16286 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:27.317612 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.318810 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.319782 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.320726 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.321671 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.321727 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.322875 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.323052 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.323653 13888 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:27.323889 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.324442 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.324883 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.326264 16286 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.298702 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:59:27.266990 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:27.214674 16425 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.322897 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.329926 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:27.330180 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.212327 16440 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:27.359015 13960 raft_consensus.cc:1238] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
W20250623 01:59:27.285236 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 149283us
I20250623 01:59:27.361611 13810 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:27.330471 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.353871 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.254715 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:27.363907 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.365692 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.366668 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.367606 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.368780 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.369441 13887 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 2.
I20250623 01:59:27.340651 13926 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:27.371397 13810 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:27.371688 13926 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:27.367724 16390 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:27.373248 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.374292 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:27.292919 16463 log.cc:927] Time spent T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.070s user 0.002s sys 0.000s
I20250623 01:59:27.250986 16322 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.263499 16444 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:27.400511 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.375185 13926 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:27.258172 16459 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.236001 16445 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.405740 14508 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:27.401068 16449 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:27.413434 16462 consensus_queue.cc:1046] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250623 01:59:27.407351 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 67652.1us
I20250623 01:59:27.409504 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.343397 16431 consensus_queue.cc:1046] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20250623 01:59:27.269150 16388 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:27.375801 16407 consensus_queue.cc:1046] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20250623 01:59:27.344113 16457 consensus_queue.cc:1046] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20250623 01:59:27.410331 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.368866 16440 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:27.415091 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.415799 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.408278 13926 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:27.440303 13960 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:27.416527 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.478843 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.452514 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:27.415936 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.427922 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.509974 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.511703 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.424270 13881 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:27.424129 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.515208 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.447642 13958 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:27.400707 16396 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:27.512975 13885 raft_consensus.cc:2391] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 9: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:27.452410 14508 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:27.425383 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.413827 16449 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:27.521334 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 113143us
I20250623 01:59:27.508257 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.417470 14479 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:27.476464 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 70173.6us
I20250623 01:59:27.522163 13926 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:27.522640 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 105333us
I20250623 01:59:27.512387 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.522926 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:27.475720 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:27.518112 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.442035 16364 raft_consensus.cc:991] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:27.525736 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.527426 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.533183 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.528975 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.529918 16455 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:27.453706 16462 consensus_queue.cc:1046] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.077s
I20250623 01:59:27.460232 16466 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:27.538201 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.524663 16449 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
W20250623 01:59:27.538709 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 126478us
I20250623 01:59:27.531917 14514 raft_consensus.cc:1238] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:27.535014 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.535144 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.419406 16360 raft_consensus.cc:3053] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:27.424106 16445 raft_consensus.cc:3053] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:27.518023 16396 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:27.536571 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.534015 13884 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:27.536667 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.416836 16286 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:27.452809 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:27.453212 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:27.531288 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.540412 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.404691 16418 consensus_queue.cc:1046] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.135s
I20250623 01:59:27.543196 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.544739 14478 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:27.524384 16389 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:27.535640 16455 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:27.553896 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:27.563567 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.550683 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.565544 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.566136 13880 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:27.538785 16466 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:27.542081 16360 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:27.420596 16438 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:27.567941 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.591096 16388 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.592192 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:27.602662 13883 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
W20250623 01:59:27.612893 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 90788us
I20250623 01:59:27.615473 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.554713 16455 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:27.550225 16431 consensus_queue.cc:1046] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20250623 01:59:27.619336 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.619163 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:27.621655 14478 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:27.588312 16466 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:27.550241 16396 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:27.546744 16445 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:27.550405 16407 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:27.616763 14507 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:27.623013 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.623752 14515 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:27.617029 14506 raft_consensus.cc:2391] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:27.594020 16477 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:27.586632 16286 consensus_queue.cc:1046] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20250623 01:59:27.594807 16470 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250623 01:59:27.635051 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:27.700383 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 138078us
I20250623 01:59:27.713829 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.663942 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.624374 16452 raft_consensus.cc:3053] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:27.753595 13888 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:27.755722 13853 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:27.760845 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.762583 13812 raft_consensus.cc:1238] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:27.763677 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.763785 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.765933 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.766523 13813 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:27.634835 16402 raft_consensus.cc:3053] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:27.770071 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.772119 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.799078 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.651485 16454 raft_consensus.cc:3053] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:27.799628 13812 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:27.799785 13813 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:27.635932 16459 raft_consensus.cc:3053] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:27.771294 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.709415 16472 raft_consensus.cc:1079] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:27.633826 16407 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:27.652189 16457 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:27.753374 16452 raft_consensus.cc:738] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:27.751185 16390 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:27.805895 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.807925 13853 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
W20250623 01:59:27.808815 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 181780us
I20250623 01:59:27.812268 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.810251 13808 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:59:27.813699 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.802958 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.821103 13812 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:27.708904 16480 consensus_queue.cc:1046] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.103s
I20250623 01:59:27.799098 16402 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:27.838521 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.838946 16402 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.839805 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:27.840204 16402 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:59:27.840364 16469 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250623 01:59:27.840965 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:27.835222 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.841714 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.837810 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:27.814739 16407 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:27.841596 16364 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:27.834249 13810 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:27.809967 16459 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:27.804889 16454 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:27.881640 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.893934 16488 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:27.894526 16488 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.757268 16477 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:27.897253 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 136848us
I20250623 01:59:27.898360 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.899021 13809 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:27.891904 13776 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:27.833559 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.904302 16476 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:27.905417 16478 raft_consensus.cc:3053] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:27.905586 13805 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:27.905838 16478 raft_consensus.cc:738] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:27.903110 13776 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:27.908649 13812 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:27.908802 13776 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:27.910236 16487 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:27.910427 13776 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:27.769901 16445 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.910439 14515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48402 (request call id 210) took 1169 ms. Trace:
I20250623 01:59:27.915206 14515 rpcz_store.cc:276] 0623 01:59:26.741066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:26.741204 (+ 138us) service_pool.cc:225] Handling call
0623 01:59:26.742819 (+ 1615us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:27.752929 (+1010110us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:27.910236 (+157307us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b540075ec54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:27.910385 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2021056384}
I20250623 01:59:27.891858 16431 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250623 01:59:27.919021 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 256794us
I20250623 01:59:27.922580 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:27.902746 16286 raft_consensus.cc:3053] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:27.939697 16485 raft_consensus.cc:3053] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:27.940140 16485 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:27.917016 16480 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:27.834100 16457 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:27.944221 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 330568us
I20250623 01:59:27.918874 16438 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:27.948248 16466 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.955840 16488 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.957608 16477 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:27.957368 16407 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.929991 16286 raft_consensus.cc:738] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:27.958922 16286 consensus_queue.cc:260] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:27.960222 16286 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:27.960983 13852 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:27.962131 13852 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:27.962759 16407 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:27.963164 13852 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:27.963371 16418 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:27.963977 16418 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:27.964546 16418 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:27.964919 13852 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:27.924731 16449 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:27.970616 16457 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:27.971751 13852 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:27.974761 13852 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:27.971661 16478 consensus_queue.cc:260] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:27.977753 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 67397.8us
I20250623 01:59:27.979339 13960 raft_consensus.cc:1238] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:28.001929 13776 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.000654 13852 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.003845 13954 raft_consensus.cc:1238] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:28.004344 13852 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.004983 13776 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.006595 13852 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.001302 13958 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:27.972476 16364 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.009850 13852 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.007735 16418 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.011241 13776 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.013018 13961 raft_consensus.cc:1238] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:28.014164 16481 raft_consensus.cc:2747] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.015009 16470 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:28.016289 13957 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
W20250623 01:59:28.016490 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 70604.5us
I20250623 01:59:27.997884 16431 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.019951 13776 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.021009 13956 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:28.022311 13776 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:27.999851 16402 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.023882 13776 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
W20250623 01:59:28.026060 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 112662us
I20250623 01:59:28.024955 13926 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.026048 13776 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.020910 16457 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.023193 16418 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:28.003816 16485 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.025535 16488 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:28.020790 16487 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.028367 16437 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:28.026764 16480 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.083217 16437 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.083678 16437 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.084378 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:28.084924 13808 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:28.086421 16449 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:28.090534 16445 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:28.091847 16478 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:28.064806 16471 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.066664 16456 raft_consensus.cc:991] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:28.069741 16486 consensus_queue.cc:1046] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:28.070613 16459 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:28.074322 16431 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:28.074921 16402 consensus_queue.cc:1046] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:28.075958 13810 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250623 01:59:28.038591 16479 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:28.077649 13925 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.077790 16396 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.078416 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.028021 16364 consensus_queue.cc:1046] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20250623 01:59:27.950723 16454 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:28.107453 13955 raft_consensus.cc:1238] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:28.039513 16499 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:28.081727 16452 consensus_queue.cc:260] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:28.109978 16426 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.110531 16452 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.110952 16396 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:28.102181 16485 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:28.111874 16360 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.112537 16396 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.114600 13925 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.115012 16454 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.115442 16500 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.116641 16500 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250623 01:59:28.117597 16492 consensus_queue.cc:1162] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "85a58a834b6b4b0dbe197b3dad87e382" responder_term: 6 status { last_received { term: 6 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 6 index: 7 } }
I20250623 01:59:28.118662 13925 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:28.119431 16483 consensus_queue.cc:1162] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ee041a8325864e788a383b8356afc652" responder_term: 6 status { last_received { term: 6 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 6 index: 7 } }
I20250623 01:59:28.120332 13925 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.120793 16483 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:28.038224 13926 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:28.121646 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 83536.2us
I20250623 01:59:28.122504 16484 raft_consensus.cc:3053] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:28.122725 13925 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.122910 16484 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:59:28.099642 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 478031us
I20250623 01:59:28.124020 16498 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:28.123822 16484 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.125083 16484 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.125435 13853 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.123811 13925 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.128322 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.130236 16364 raft_consensus.cc:3053] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:28.130679 16364 raft_consensus.cc:738] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:59:28.101909 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 391327us
I20250623 01:59:28.130882 16472 raft_consensus.cc:3053] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:28.131279 16431 consensus_queue.cc:1046] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:28.131738 16449 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.131495 16364 consensus_queue.cc:260] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.131258 16472 raft_consensus.cc:738] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:28.132412 16431 raft_consensus.cc:3053] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:28.132654 16364 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.132771 16431 raft_consensus.cc:738] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:28.108068 16456 raft_consensus.cc:1079] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:28.132900 16472 consensus_queue.cc:260] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.133615 16431 consensus_queue.cc:260] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.134186 16472 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:28.134786 16459 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:28.135897 13853 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.037577 16482 consensus_queue.cc:1046] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20250623 01:59:28.136744 16486 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.038681 16491 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.107877 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.137557 16482 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.110548 16437 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.137761 16491 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:28.138854 16462 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.139912 16437 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.121007 16500 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:28.121714 16396 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250623 01:59:28.130273 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:28.141533 16402 raft_consensus.cc:3053] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:28.141927 16402 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:28.142704 16402 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.144009 16402 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.146049 13925 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.113035 16360 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:28.017575 16440 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:28.150483 16500 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:28.150875 16500 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:28.151238 16500 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.149369 13925 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
W20250623 01:59:28.158012 16455 consensus_meta.cc:287] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.532s user 0.004s sys 0.005s
W20250623 01:59:28.098434 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 89484.5us
I20250623 01:59:28.098649 13776 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.159096 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:28.134927 16431 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.166747 13925 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.201653 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.203474 16498 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.204447 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.206496 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.207149 13959 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 9: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:28.208771 16486 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250623 01:59:28.211643 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:28.213397 13925 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:28.162186 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 63611.2us
I20250623 01:59:28.218525 13776 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.219741 13776 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.221405 13776 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.222090 16479 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:28.222363 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.222669 16479 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:59:28.223043 13812 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:28.223039 16479 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.223670 16479 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:28.223788 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.224833 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:28.226193 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2928af9e5df4f9eacf59f499ab96dc1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:28.226526 16472 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.231583 16452 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.231751 16483 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.165650 16476 consensus_queue.cc:1046] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20250623 01:59:28.170404 16488 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:28.172864 16499 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.176486 14506 raft_consensus.cc:1238] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:28.177105 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:28.177297 14479 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.244804 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.182590 16492 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:28.245419 16386 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.189869 16466 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:28.166409 16455 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.191009 13807 raft_consensus.cc:1238] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:28.192005 16440 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:28.194197 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.248124 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:28.248593 13958 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:28.248656 13956 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:28.249135 13958 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:28.249603 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.250222 13806 raft_consensus.cc:2391] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 9: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:28.250303 13956 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 9: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:28.251145 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.252017 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:28.248502 16455 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:28.252918 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:28.194626 13808 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:28.255569 16482 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:28.256919 13779 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.195948 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.198012 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.196410 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.258337 13804 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 9: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:28.200313 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:28.200455 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.200938 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:28.259274 13810 raft_consensus.cc:2391] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 9: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:28.214469 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.259775 13959 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:28.260118 16476 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Stepping down as leader of term 8
W20250623 01:59:28.260653 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:28.227087 16472 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:28.262415 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.262867 16472 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:28.232829 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.263475 13812 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.263432 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.264273 16471 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.264760 16462 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.264832 16471 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.216378 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.267688 13958 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.269150 13953 raft_consensus.cc:1238] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 6. Ops: []
I20250623 01:59:28.159096 13852 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.235543 16498 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:28.269958 16386 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:28.270200 16498 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:28.270495 16386 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:28.272380 16386 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.237385 16491 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:28.149143 13813 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:28.237417 13925 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.238302 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:28.274329 16455 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.274955 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:28.275657 13960 raft_consensus.cc:2391] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 5: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:28.260509 16476 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:28.275483 13925 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.277067 16476 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:28.278849 16476 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:28.280071 13958 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:28.241530 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:28.284647 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:28.285017 16466 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:28.285367 13961 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:28.241803 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:28.244853 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 67666us
I20250623 01:59:28.286139 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:28.287120 13925 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.232735 16479 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.201534 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:28.248531 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.289404 16364 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:28.289947 16479 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.270778 16498 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:59:28.288416 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 129424us
I20250623 01:59:28.291343 13852 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.274406 14511 raft_consensus.cc:2391] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:28.178634 14512 raft_consensus.cc:1238] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:28.292099 16364 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.292865 16364 consensus_queue.cc:1046] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:28.293843 16364 raft_consensus.cc:3053] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:28.294188 16364 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:28.296816 13806 raft_consensus.cc:1238] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:59:28.234817 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:28.297866 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 63115.6us
I20250623 01:59:28.298084 16364 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.298575 13776 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.299314 16488 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:28.299882 16488 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.300750 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:28.302167 16364 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:28.235920 16500 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.302898 13954 raft_consensus.cc:1238] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:28.303913 16500 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:28.303997 13882 raft_consensus.cc:1238] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:28.305061 13852 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.305913 16360 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.306252 13852 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.306972 16360 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.292243 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:28.307713 13852 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.308243 13808 raft_consensus.cc:2391] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 5: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:28.308418 16360 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.308835 13852 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.311580 16360 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:28.311939 13852 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.312630 16360 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:28.313027 16360 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.313385 13852 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.313478 16360 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:28.313539 14479 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.315322 16461 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.325145 16466 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250623 01:59:28.327419 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:28.327566 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:28.330580 16498 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.332476 16498 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:28.310916 13959 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:28.336920 13776 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.337087 13812 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:28.338377 13852 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.338747 13883 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:28.339737 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.341064 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:28.345006 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.346678 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.347455 16452 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:28.349964 13884 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250623 01:59:28.350426 16472 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.363638 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52444 (request call id 274) took 2735 ms. Trace:
W20250623 01:59:28.357569 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 98147.4us
I20250623 01:59:28.364626 13926 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.365603 16454 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.366025 13776 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.367125 13926 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.368260 13926 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.369261 16498 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:28.373203 16488 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.373654 16500 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.369117 16499 consensus_queue.cc:1046] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20250623 01:59:28.376008 16499 raft_consensus.cc:3053] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:28.376360 16499 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.360363 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:28.377524 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.379892 13926 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.381847 13852 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.392573 13926 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:28.394997 13852 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.397289 13852 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.402220 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.413921 13926 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:28.387512 16455 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.388775 16364 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:28.364045 13710 rpcz_store.cc:276] 0623 01:59:25.628574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:25.628725 (+ 151us) service_pool.cc:225] Handling call
0623 01:59:25.914208 (+285483us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54003868d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e97bafac 00007f41e98b37ee 00007f41e7dfd624 00007f41e7dfd5b2 00007f41e7dfd4e1 00007f41e7dfd45c 00007f41e7dfc712 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:28.363575 (+2449367us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:27.408416 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:27.408987 (+ 571us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:27.409020 (+ 33us) write_op.cc:435] Acquired schema lock
0623 01:59:27.409040 (+ 20us) tablet.cc:660] Decoding operations
0623 01:59:27.412733 (+ 3693us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:27.412795 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:27.412826 (+ 31us) tablet.cc:683] Acquiring locks for 2 operations
0623 01:59:27.413155 (+ 329us) tablet.cc:699] Row locks acquired
0623 01:59:27.413174 (+ 19us) write_op.cc:260] PREPARE: finished
0623 01:59:27.413336 (+ 162us) write_op.cc:270] Start()
0623 01:59:27.413504 (+ 168us) write_op.cc:276] Timestamp: P: 1750643967413266 usec, L: 0
0623 01:59:27.413526 (+ 22us) op_driver.cc:348] REPLICATION: starting
0623 01:59:27.547327 (+133801us) log.cc:844] Serialized 907 byte log entry
0623 01:59:28.045892 (+498565us) op_driver.cc:464] REPLICATION: finished
0623 01:59:28.328995 (+283103us) write_op.cc:301] APPLY: starting
0623 01:59:28.329114 (+ 119us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:28.329242 (+ 128us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:28.329409 (+ 167us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:28.329854 (+ 445us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:28.330122 (+ 268us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 01:59:28.330187 (+ 65us) write_op.cc:312] APPLY: finished
0623 01:59:28.330420 (+ 233us) log.cc:844] Serialized 29 byte log entry
0623 01:59:28.330582 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:28.330669 (+ 87us) write_op.cc:454] Released schema lock
0623 01:59:28.330709 (+ 40us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":91224,"reported_tablets":5,"spinlock_wait_cycles":631340288,"child_traces":[["op",{"apply.queue_time_us":282821,"apply.run_cpu_time_us":2201,"apply.run_wall_time_us":60078,"num_ops":2,"prepare.queue_time_us":367432,"prepare.run_cpu_time_us":3261,"prepare.run_wall_time_us":152023,"replication_time_us":617721,"thread_start_us":221921,"threads_started":2}]]}
I20250623 01:59:28.415851 13852 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.417765 13779 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.419735 13852 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.391415 16483 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.421908 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.423276 13779 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.425092 13779 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.425432 13959 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "afad1ff768904b11936d338d33df7998"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:28.394621 16472 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:28.428571 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 253) took 1459 ms. Trace:
I20250623 01:59:28.428929 13711 rpcz_store.cc:276] 0623 01:59:26.953815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:26.962529 (+ 8714us) service_pool.cc:225] Handling call
0623 01:59:28.412343 (+1449814us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14958,"reported_tablets":5}
I20250623 01:59:28.429738 16472 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.430148 16472 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.431641 16488 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.420013 16364 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.433377 16364 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.436046 16482 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.438024 13852 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.426214 13959 raft_consensus.cc:604] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:28.440687 16482 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:28.441967 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.443454 16472 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.421293 16461 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.446609 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.448076 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.449298 13962 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:28.451117 16472 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.418684 16492 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.418831 16466 consensus_queue.cc:1046] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20250623 01:59:28.455160 16492 raft_consensus.cc:3053] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:28.455514 16492 raft_consensus.cc:738] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.456246 16492 consensus_queue.cc:260] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.458161 16492 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:28.387104 16360 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.412346 16396 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:28.415544 16431 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:59:28.476661 13852 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:28.472836 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 47807us
I20250623 01:59:28.477625 13779 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.424717 16484 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:28.460812 16491 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.421577 16402 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:28.503105 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.483314 13852 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.484345 13779 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.426651 16449 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:28.424389 16459 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.488696 16360 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.476594 16431 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.410990 16499 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.483529 16396 raft_consensus.cc:695] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.491971 13954 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:28.486847 16484 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.515769 16484 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.515762 13954 raft_consensus.cc:604] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.516028 16396 consensus_queue.cc:237] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:28.486217 16445 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.483795 16472 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:28.520464 16472 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.520840 16472 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.524299 16499 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.507486 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.476867 16476 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.513957 13805 raft_consensus.cc:2391] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 5: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:28.514043 16402 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:28.514226 16449 raft_consensus.cc:695] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.528580 16449 consensus_queue.cc:237] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.514560 16431 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:28.526894 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.527304 16402 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.538795 16402 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.527710 16445 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.546686 16402 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.476971 16364 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.557667 16445 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.560528 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.571172 13883 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:28.571579 13883 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.572731 13779 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.573514 16476 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.563740 16484 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.515292 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.577178 16445 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.580109 13883 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:28.580893 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 65684.2us
I20250623 01:59:28.583528 13852 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.582405 16360 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:28.584332 16360 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.584702 16360 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.579134 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.581722 16482 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.598836 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:28.600327 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:28.600941 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:28.601732 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.602579 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.602840 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.603870 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.601431 13885 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:28.605489 13879 raft_consensus.cc:2366] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6.
I20250623 01:59:28.605899 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.607046 13925 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.594708 16445 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:28.610976 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:28.608187 13925 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.608428 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.608847 13806 raft_consensus.cc:1238] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:28.595491 16485 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:28.612113 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.612645 13810 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:28.614126 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.614133 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.617583 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:28.618785 13882 raft_consensus.cc:1238] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:28.619764 13882 raft_consensus.cc:1238] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:28.620682 13882 raft_consensus.cc:1238] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:28.621428 16452 consensus_queue.cc:1046] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:28.621742 13884 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:28.622386 16452 raft_consensus.cc:3053] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:28.622742 16452 raft_consensus.cc:738] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:28.623153 16418 consensus_queue.cc:1046] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:28.623497 16452 consensus_queue.cc:260] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.624190 16501 raft_consensus.cc:3053] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:28.624548 16501 raft_consensus.cc:738] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:28.624719 16452 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.625277 16501 consensus_queue.cc:260] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.599346 16486 consensus_queue.cc:1046] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20250623 01:59:28.626772 16501 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.608729 16491 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:28.628444 13954 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:28.630213 13883 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.610482 13805 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:28.633167 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.633699 13954 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.635474 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.635910 13852 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.611552 13808 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:28.636121 13805 raft_consensus.cc:3053] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:28.636857 13805 raft_consensus.cc:738] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:28.633409 16501 consensus_queue.cc:1046] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250623 01:59:28.637620 16498 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:28.641392 13880 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:28.645064 13886 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:28.647642 13886 raft_consensus.cc:1238] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:28.650590 13888 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:28.654202 13880 raft_consensus.cc:1238] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
W20250623 01:59:28.654395 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 46279.7us
I20250623 01:59:28.610608 13779 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:28.654763 13810 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:28.655608 13925 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:28.660778 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 50214us
I20250623 01:59:28.661754 13779 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.637744 13805 consensus_queue.cc:260] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.663380 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:28.664206 13925 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.666332 13805 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:28.672837 13925 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.658628 16396 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:28.665271 16445 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:28.666165 16454 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:28.665788 16476 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:59:28.676576 13925 leader_election.cc:400] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6.
I20250623 01:59:28.681027 13925 leader_election.cc:403] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20250623 01:59:28.679013 13853 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:28.681241 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 44095.5us
I20250623 01:59:28.681931 13852 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:28.682943 13852 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.683432 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.685070 13925 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:28.675220 16498 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.676088 16499 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:28.692690 13954 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:28.686379 16445 raft_consensus.cc:695] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:28.698589 16484 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.686192 16396 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6.
I20250623 01:59:28.711020 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.710808 16484 raft_consensus.cc:2802] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:28.710870 16445 consensus_queue.cc:237] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.711954 16484 raft_consensus.cc:695] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:28.711966 13854 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:28.704804 16508 log.cc:927] Time spent T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20250623 01:59:28.711290 16499 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:28.702766 16482 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:28.714361 16482 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:28.713701 16484 consensus_queue.cc:237] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.718544 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.719280 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.718925 13954 raft_consensus.cc:1238] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:28.703832 16360 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.721570 13852 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.723459 13884 raft_consensus.cc:1238] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:28.704164 16364 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.688601 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.707437 13885 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:28.708262 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.709069 13805 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:28.673152 16459 consensus_queue.cc:1046] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20250623 01:59:28.714784 16482 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.684177 16402 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.726572 13852 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.685408 16501 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.734871 13805 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f44146f651124d8fba4cc1825f480603"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:28.732712 13854 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.735739 13805 raft_consensus.cc:604] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:28.736322 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.736637 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.736889 13808 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:28.737625 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.741844 13813 raft_consensus.cc:1238] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:28.744925 16513 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:28.745162 16396 consensus_queue.cc:1046] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:28.745462 16513 raft_consensus.cc:695] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:28.745728 16360 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.746181 16396 raft_consensus.cc:3053] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:28.746592 16396 raft_consensus.cc:738] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.740935 16499 consensus_queue.cc:1046] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20250623 01:59:28.747573 16501 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:28.747432 16396 consensus_queue.cc:260] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.748569 16396 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.746306 16360 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.764317 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.687887 16431 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.765961 16449 consensus_queue.cc:1046] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:28.763068 13925 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.747432 16499 raft_consensus.cc:3053] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:28.750931 16459 raft_consensus.cc:3053] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:28.751133 16454 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:28.773945 16466 raft_consensus.cc:2802] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:28.776505 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250623 01:59:28.759810 16418 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.780476 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:28.781469 13881 raft_consensus.cc:1238] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 2. Current term is 3. Ops: []
I20250623 01:59:28.779516 13925 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.781857 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:28.746363 16513 consensus_queue.cc:237] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.791635 16466 raft_consensus.cc:695] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.792503 16466 consensus_queue.cc:237] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.771836 16459 raft_consensus.cc:738] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:28.689069 16486 consensus_queue.cc:1046] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20250623 01:59:28.799170 13925 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:28.804464 13806 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:28.769373 16499 raft_consensus.cc:738] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.847317 16454 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:28.847872 16454 raft_consensus.cc:695] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.848747 16454 consensus_queue.cc:237] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.850492 16493 consensus_queue.cc:1046] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:28.854331 16431 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.856035 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.841282 16455 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
W20250623 01:59:28.844965 16507 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.251s user 0.001s sys 0.000s
I20250623 01:59:28.875975 16459 consensus_queue.cc:260] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.892557 16499 consensus_queue.cc:260] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.893919 16499 raft_consensus.cc:3058] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:28.903139 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:28.908389 13808 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:28.905256 16455 raft_consensus.cc:695] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:28.928054 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 79632.9us
I20250623 01:59:28.928825 13926 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:28.929792 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:28.930549 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 204036us
I20250623 01:59:28.932144 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.932798 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.938469 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.939561 13852 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.940925 16431 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.941689 16431 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:59:28.944130 16455 consensus_queue.cc:237] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.939347 16454 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:28.950964 16472 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:28.952823 16472 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:28.953452 13883 raft_consensus.cc:1238] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:28.955653 16514 raft_consensus.cc:3053] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:28.955869 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:28.956024 16514 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:28.959896 13959 raft_consensus.cc:1238] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:28.962172 16459 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:28.966979 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:28.969942 13926 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:28.971046 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 171955us
I20250623 01:59:28.972211 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:28.972321 13926 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:28.975387 16500 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:28.975643 16449 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:28.976868 16518 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:28.979269 16500 raft_consensus.cc:695] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.982149 16500 consensus_queue.cc:237] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.002489 16444 consensus_queue.cc:1046] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250623 01:59:28.979959 16521 consensus_queue.cc:1046] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20250623 01:59:29.001602 16440 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:29.014994 16517 log.cc:927] Time spent T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.126s user 0.004s sys 0.005s
I20250623 01:59:29.014901 16440 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.016945 16440 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.021749 16483 raft_consensus.cc:3053] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:29.022174 16483 raft_consensus.cc:738] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:29.023010 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:29.023569 13883 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:29.024787 13776 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.000373 16519 consensus_queue.cc:1046] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20250623 01:59:28.995026 16449 raft_consensus.cc:695] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:29.033783 16452 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:29.037096 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.037739 13954 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:29.039012 16452 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.039451 16452 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:28.997211 16482 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.022886 16483 consensus_queue.cc:260] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.046111 16478 raft_consensus.cc:3053] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:29.046334 16454 raft_consensus.cc:3053] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:29.047236 16454 raft_consensus.cc:738] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:28.995620 16514 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.048007 16454 consensus_queue.cc:260] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.048784 16514 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:29.049269 16454 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:29.052206 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:29.052959 14507 raft_consensus.cc:2391] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 4: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:29.046489 16478 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:29.053620 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:29.054287 16478 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:29.055552 16478 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:29.069207 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:29.071797 13808 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:29.070873 16483 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:29.074306 13711 catalog_manager.cc:5582] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:29.077935 16520 raft_consensus.cc:3053] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:29.091758 16474 consensus_queue.cc:1046] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:29.092176 16520 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:29.082839 13709 catalog_manager.cc:5582] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 5 to 6, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:29.092736 16474 consensus_queue.cc:1046] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:29.048009 16482 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:29.094552 13711 catalog_manager.cc:5582] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 8, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:29.092973 16520 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.105386 16520 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:29.105754 16492 raft_consensus.cc:3053] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:29.106462 16492 raft_consensus.cc:738] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:29.096868 13806 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250623 01:59:29.108170 16492 consensus_queue.cc:260] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.107487 16483 raft_consensus.cc:3053] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:29.107668 16474 consensus_queue.cc:1046] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250623 01:59:29.097048 13954 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:29.111783 13854 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:29.113224 16449 consensus_queue.cc:237] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:29.109707 16455 consensus_queue.cc:1162] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "85a58a834b6b4b0dbe197b3dad87e382" responder_term: 6 status { last_received { term: 6 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 6 index: 3 } }
I20250623 01:59:29.109385 16492 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:29.119608 16482 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:29.120055 16482 raft_consensus.cc:695] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:29.120769 16482 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.121809 16474 raft_consensus.cc:3053] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:29.110119 16483 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:29.122170 16474 raft_consensus.cc:738] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:29.122771 16483 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.126695 13883 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:29.128082 16466 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:29.129439 16483 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:29.127031 16527 raft_consensus.cc:991] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:29.122891 16474 consensus_queue.cc:260] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.135411 16474 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:29.136597 16452 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.144160 14507 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250623 01:59:29.163398 13806 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:29.154568 16527 raft_consensus.cc:1079] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:29.178382 13812 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:29.168795 16500 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20250623 01:59:29.181537 13881 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:29.184334 16452 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.185909 16440 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:29.187620 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:29.188041 13887 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:29.194623 13887 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:29.196586 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:29.197009 13959 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:29.165220 16454 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20250623 01:59:29.200083 16466 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:29.199262 16514 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20250623 01:59:29.209594 13776 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.218061 16462 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:29.219362 13959 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:29.219786 13806 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48570
I20250623 01:59:29.220573 13806 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:29.221009 13806 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:29.218531 16462 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:29.232393 13806 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:29.234720 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2928af9e5df4f9eacf59f499ab96dc1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:29.236608 13806 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.222478 16479 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.241984 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:29.251734 13958 raft_consensus.cc:1273] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:29.262333 13954 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:29.261884 16482 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
W20250623 01:59:29.257097 16531 log.cc:927] Time spent T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20250623 01:59:29.277673 13710 catalog_manager.cc:5582] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 4 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 5 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:29.285956 16476 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:29.286684 16438 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.287252 16438 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.289675 16491 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.290241 16514 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.290498 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:29.290791 16514 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.275929 16462 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.292850 16514 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.294214 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.247601 16479 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.294831 13954 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:29.295019 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:29.295732 13884 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:29.297235 13810 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:29.298501 13962 raft_consensus.cc:1273] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:29.299733 13959 raft_consensus.cc:3053] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:29.300186 13959 raft_consensus.cc:738] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:29.302471 16479 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:29.303078 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:29.304138 16438 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.304265 13806 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:29.304987 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.306412 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:29.307859 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.308848 13962 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:29.310359 14479 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.310631 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.312481 13806 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:29.301043 13959 consensus_queue.cc:260] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.290166 16491 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.317250 13959 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:59:29.320923 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 92067.4us
I20250623 01:59:29.314805 13955 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:29.331651 16491 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:29.332471 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:29.333077 13884 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:29.324601 14511 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:29.282097 16482 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.333873 16438 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:29.334196 16482 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.335305 16438 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.335721 16438 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:29.336143 16482 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.337682 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:29.338310 14511 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:29.339915 13779 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:29.340955 16491 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:29.341347 16491 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.341691 16491 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:29.343914 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:29.344509 13887 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:29.344601 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.346441 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:29.347123 13808 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:29.347582 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:29.348162 13881 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:29.349997 13959 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:29.350114 13852 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.350742 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.353005 13884 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:29.357631 13955 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:29.353724 13806 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:29.361342 13958 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:29.345422 16478 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:29.388753 13887 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:29.389003 16478 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.389437 16478 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:29.374833 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:29.377144 13776 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.378367 16482 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:29.390671 16482 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.390880 16418 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.391093 16482 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:29.391460 16418 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.393455 16418 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.393836 16452 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:29.346101 16514 consensus_queue.cc:1035] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20250623 01:59:29.394294 16452 raft_consensus.cc:695] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:29.395133 16452 consensus_queue.cc:237] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.396152 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.396710 13958 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:29.397238 13807 raft_consensus.cc:3058] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:29.399523 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:29.400117 13808 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:29.401468 13854 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.404074 16500 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:29.404666 16479 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:29.405126 16479 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.405520 16479 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:29.404758 13812 raft_consensus.cc:1273] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:29.406239 13887 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:29.421260 16438 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.422122 13887 raft_consensus.cc:1273] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:29.429266 16438 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.432351 16478 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.434065 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:29.434643 13958 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 8
I20250623 01:59:29.435590 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:29.436355 13887 raft_consensus.cc:3058] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:29.417665 16521 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:29.462026 13887 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
I20250623 01:59:29.467013 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:29.409700 16466 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20250623 01:59:29.417557 16484 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250623 01:59:29.473891 13958 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
I20250623 01:59:29.475971 14481 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.455942 16521 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.492588 16521 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:29.442414 16483 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250623 01:59:29.485199 16384 raft_consensus.cc:2802] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:29.443368 16492 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20250623 01:59:29.505863 16478 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.513018 13811 raft_consensus.cc:1273] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:29.473851 16485 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.521467 16485 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.523767 16384 raft_consensus.cc:695] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:29.523417 16525 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:29.526813 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:29.527596 13958 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:29.537739 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:29.538569 13813 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:29.537817 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:29.540174 13811 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:29.542045 13852 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.524835 16384 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.523468 16485 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.548065 13813 raft_consensus.cc:1273] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:29.554319 13811 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:29.465504 16491 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.554674 16514 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:29.569583 16488 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.555974 13852 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.580343 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.580904 13887 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:29.581251 13955 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:29.573750 16493 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20250623 01:59:29.589836 16488 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.591593 16484 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:29.591944 16488 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:29.592125 16484 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:29.594916 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:29.599527 13887 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:29.583534 16514 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.626688 16514 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:29.594259 16525 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.632622 13958 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:29.639245 13804 raft_consensus.cc:2391] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 8: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
W20250623 01:59:29.629739 16547 log.cc:927] Time spent T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20250623 01:59:29.652467 13807 raft_consensus.cc:1273] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:29.665536 13887 raft_consensus.cc:1273] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:29.630142 16485 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:29.641381 16438 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.593226 16484 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.610016 16454 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20250623 01:59:29.664022 16459 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.596866 16479 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.675801 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:29.651149 16525 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:29.730837 13887 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:29.674723 16485 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.734825 16476 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:29.729538 14511 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:29.749501 16472 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.750164 16472 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.678716 16492 consensus_queue.cc:1035] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20250623 01:59:29.701795 16483 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:29.794764 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 127886us
I20250623 01:59:29.794961 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:29.795693 13885 raft_consensus.cc:2391] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 8: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:29.799818 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:29.800431 13812 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:29.792136 13805 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:59:29.792673 13958 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:59:29.692184 16459 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.793881 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:29.814127 13959 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:29.678092 16438 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:29.801162 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 125375us
I20250623 01:59:29.808030 13852 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:29.797014 16493 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.790864 16472 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:29.849071 13807 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:29.792635 16466 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20250623 01:59:29.818833 16518 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:29.792285 16482 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:29.877638 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 69707.2us
W20250623 01:59:29.935029 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 55234.5us
I20250623 01:59:29.836427 16478 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20250623 01:59:29.799742 16483 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:29.940716 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 62006.6us
I20250623 01:59:29.944131 13778 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:29.876276 16472 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:29.876967 16551 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:29.871752 16493 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:29.961627 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 93269.6us
I20250623 01:59:29.964010 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:29.951200 16472 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:29.949584 16491 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:29.968904 13955 raft_consensus.cc:2391] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 8: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:29.877167 16518 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:29.959518 16445 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:29.975384 16472 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:29.958086 16551 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:30.006597 14507 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:59:30.008785 16485 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:30.009896 16556 log.cc:927] Time spent T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20250623 01:59:30.011242 16459 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.018543 16493 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.019213 16438 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:30.019974 16554 log.cc:927] Time spent T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.204s user 0.000s sys 0.000s
I20250623 01:59:30.021986 16483 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.023561 16518 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:30.011242 16557 log.cc:927] Time spent T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
W20250623 01:59:30.028308 16521 consensus_meta.cc:287] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.535s user 0.009s sys 0.000s
W20250623 01:59:30.029721 16552 log.cc:927] Time spent T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20250623 01:59:30.030354 16525 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:29.994635 16445 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.004801 16570 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
I20250623 01:59:30.038566 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:29.999091 16478 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:30.003764 16560 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20250623 01:59:30.088634 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.090440 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:30.130308 13808 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:30.143069 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.144137 13810 raft_consensus.cc:2391] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:30.036943 16445 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
W20250623 01:59:30.150251 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 185093us
I20250623 01:59:30.151423 13779 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:30.129666 16540 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.155929 13808 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
W20250623 01:59:30.156175 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 212249us
I20250623 01:59:30.035635 16521 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.157927 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.157076 13852 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.159529 13808 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:30.143589 16479 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:30.162730 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.163537 13812 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:30.168857 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.169521 13885 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:30.169833 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.171363 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.173177 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:30.173771 14511 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:30.174160 16581 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.174669 16581 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:30.158099 16540 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:30.177548 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.178117 13810 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:30.178712 16540 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.154052 16482 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.146456 16478 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.157027 16488 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:30.170604 13883 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:30.189628 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.183449 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.191588 13959 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:30.191630 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.199172 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.199828 13880 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:59:30.206413 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.207011 13811 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:30.208272 13806 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:30.214341 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:30.215093 13958 raft_consensus.cc:3058] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:30.181775 16452 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.214972 13926 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:30.221572 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:30.224758 13962 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:30.224191 13926 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.220641 13854 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:30.209903 16492 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.232120 16492 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.232503 14511 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:30.233552 16452 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.235558 16452 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.242309 13961 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:30.233906 16500 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:30.184809 16572 raft_consensus.cc:2802] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:30.246544 13926 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:30.249226 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.260569 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.261216 13961 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:30.262755 13854 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:30.268399 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 45367.7us
I20250623 01:59:30.270462 13958 raft_consensus.cc:2466] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:30.271714 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52432 (request call id 261) took 1193 ms. Trace:
I20250623 01:59:30.274505 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:30.321631 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.272190 13709 rpcz_store.cc:276] 0623 01:59:29.077951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:29.078096 (+ 145us) service_pool.cc:225] Handling call
0623 01:59:30.271634 (+1193538us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:29.172199 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:29.197720 (+ 25521us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:29.197755 (+ 35us) write_op.cc:435] Acquired schema lock
0623 01:59:29.197767 (+ 12us) tablet.cc:660] Decoding operations
0623 01:59:29.198091 (+ 324us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:29.198168 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:29.198198 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:59:29.198372 (+ 174us) tablet.cc:699] Row locks acquired
0623 01:59:29.198390 (+ 18us) write_op.cc:260] PREPARE: finished
0623 01:59:29.198525 (+ 135us) write_op.cc:270] Start()
0623 01:59:29.198644 (+ 119us) write_op.cc:276] Timestamp: P: 1750643969198474 usec, L: 0
0623 01:59:29.198661 (+ 17us) op_driver.cc:348] REPLICATION: starting
0623 01:59:29.218771 (+ 20110us) log.cc:844] Serialized 535 byte log entry
0623 01:59:29.651661 (+432890us) op_driver.cc:464] REPLICATION: finished
0623 01:59:29.945554 (+293893us) write_op.cc:301] APPLY: starting
0623 01:59:29.952170 (+ 6616us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:29.959300 (+ 7130us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:29.976378 (+ 17078us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:30.031579 (+ 55201us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:30.037236 (+ 5657us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:59:30.038522 (+ 1286us) write_op.cc:312] APPLY: finished
0623 01:59:30.038719 (+ 197us) log.cc:844] Serialized 23 byte log entry
0623 01:59:30.132163 (+ 93444us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:30.142994 (+ 10831us) write_op.cc:454] Released schema lock
0623 01:59:30.146243 (+ 3249us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:30.158843 (+ 12600us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:30.160694 (+ 1851us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":271798,"apply.run_cpu_time_us":7960,"apply.run_wall_time_us":219389,"num_ops":1,"prepare.queue_time_us":59448,"prepare.run_cpu_time_us":2755,"prepare.run_wall_time_us":59873,"replication_time_us":404138,"thread_start_us":236481,"threads_started":2}]]}
W20250623 01:59:30.295624 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 44306.2us
I20250623 01:59:30.331612 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.332221 13961 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:30.246563 16500 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.337587 13885 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:30.254722 16551 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:30.319206 16514 consensus_meta.cc:287] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.692s user 0.012s sys 0.000s
I20250623 01:59:30.266547 16527 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:30.273838 16484 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:59:30.269536 16483 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:30.311709 16581 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.361811 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.409085 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.410540 13779 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:30.363170 13954 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250623 01:59:30.364707 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:30.411065 13880 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:30.294479 16572 raft_consensus.cc:695] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:30.411552 13808 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 5
I20250623 01:59:30.261883 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:30.418931 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.351748 16452 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:30.351148 16476 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:30.361795 16527 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.341066 16500 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:30.244800 16568 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.099s
I20250623 01:59:30.310436 16492 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.311072 16575 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:30.369517 13960 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:30.311096 16472 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.316190 16445 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:30.461444 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.400876 16418 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:30.367878 16514 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:30.425552 13853 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.003s sys 0.012s
I20250623 01:59:30.397542 16483 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.366461 16484 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:30.429644 13779 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:30.492620 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 63060.3us
I20250623 01:59:30.428238 16527 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:30.423967 16540 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250623 01:59:30.489558 13853 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.624s user 0.003s sys 0.012s
I20250623 01:59:30.429749 14507 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:30.433065 16521 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:30.504122 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 75200.3us
I20250623 01:59:30.505410 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:30.505879 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.506084 13954 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:30.506500 13886 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:30.423804 16551 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20250623 01:59:30.494562 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.496122 13805 raft_consensus.cc:2366] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 9. Current term is 10.
I20250623 01:59:30.497272 13808 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:30.424993 16462 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20250623 01:59:30.510679 13888 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:30.520638 13852 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.521595 13954 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
W20250623 01:59:30.485807 16550 log.cc:927] Time spent T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.069s user 0.003s sys 0.003s
I20250623 01:59:30.512777 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.513449 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:30.531615 13884 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:30.504319 16571 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:30.366789 13925 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:30.538972 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 172260us
I20250623 01:59:30.429767 16476 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.507401 16540 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.508179 16570 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.527391 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.552443 13954 raft_consensus.cc:2391] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:30.491984 16440 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:30.497270 16484 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:30.493490 16483 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:30.553097 16525 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.551277 16571 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.554667 16587 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
I20250623 01:59:30.578763 16445 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:30.584244 16570 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.586006 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:30.586462 16570 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.586771 13954 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:30.538872 16572 consensus_queue.cc:237] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.529284 16478 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.589356 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.558799 16440 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.558871 16521 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.591784 16521 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.592784 16440 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.594133 16521 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.550871 16540 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
W20250623 01:59:30.559293 16586 log.cc:927] Time spent T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250623 01:59:30.563952 16578 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.597465 16578 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.600062 16578 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.601814 13957 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:30.565400 16562 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:30.603106 16562 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.603431 16562 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:30.604588 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:30.605415 13955 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:30.609506 13854 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.611256 16478 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:30.611752 16478 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.612141 16478 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:30.570160 16472 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.535166 13887 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:30.536972 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.499884 16459 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:30.617015 16459 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.617509 16459 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:30.620440 16527 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:30.527205 13962 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:30.435863 16452 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.622983 16452 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:30.624557 16589 log.cc:927] Time spent T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.138s user 0.000s sys 0.003s
I20250623 01:59:30.570312 16571 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.535965 13960 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:30.465747 16575 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.558919 16518 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
W20250623 01:59:30.570974 16554 log.cc:927] Time spent T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250623 01:59:30.630250 16575 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.576206 16514 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.630915 16518 raft_consensus.cc:695] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:30.565348 16476 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.568135 16525 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.631778 16518 consensus_queue.cc:237] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.634425 16525 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.636183 13854 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:30.638777 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 80651.5us
I20250623 01:59:30.639655 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:30.640211 13807 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 9
I20250623 01:59:30.573608 16471 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.643963 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.644683 13880 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:30.646031 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:30.646373 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.646602 13808 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 5
I20250623 01:59:30.647516 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.648591 13954 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:30.650665 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.646952 13806 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
W20250623 01:59:30.662695 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 152678us
I20250623 01:59:30.663388 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.547742 13884 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:30.637320 16472 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:30.666159 16472 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:30.666597 16472 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:30.665189 13925 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.663705 14481 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:30.672974 13808 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
W20250623 01:59:30.675241 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 98811.2us
I20250623 01:59:30.675419 13852 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.683876 13807 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:30.687047 13961 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:30.687686 13960 raft_consensus.cc:604] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:30.688159 13960 raft_consensus.cc:612] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:30.641782 16476 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:30.697214 13852 leader_election.cc:400] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 9. Current term is 10.
I20250623 01:59:30.700814 13852 leader_election.cc:403] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:30.676496 16587 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:30.705103 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.707454 16459 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.703399 13776 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:30.712782 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.715356 13806 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:30.715782 16588 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:30.716238 13776 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.716917 16601 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:30.717589 16601 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:30.718021 16601 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:30.689720 16551 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
W20250623 01:59:30.721979 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 292642us
I20250623 01:59:30.699218 16492 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:30.741110 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.793567 13810 raft_consensus.cc:2366] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5.
I20250623 01:59:30.741880 13807 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:30.702801 16452 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:30.718966 16601 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:30.740399 16492 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:59:30.830355 16593 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20250623 01:59:30.830583 16492 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:30.830986 16580 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.831490 16580 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.832638 16452 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.712550 16488 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
W20250623 01:59:30.834686 16600 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20250623 01:59:30.835196 16452 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.670588 16471 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.836252 13805 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:30.836809 13811 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:30.837229 16471 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.837777 16483 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.839043 16527 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:30.839665 16483 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.839742 16585 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:30.846971 16572 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:30.875763 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 182092us
I20250623 01:59:30.710146 16476 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.841691 16598 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.792563 16551 raft_consensus.cc:695] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:30.890394 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:59:30.885150 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 168963us
I20250623 01:59:30.845393 16418 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 10
I20250623 01:59:30.840945 16478 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.910750 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:30.917670 13879 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:30.919298 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.921367 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.898432 16527 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.924660 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.921815 16558 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:30.923295 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.926903 13883 raft_consensus.cc:2391] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:30.923866 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.927984 13880 raft_consensus.cc:2391] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:30.898602 16560 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.928807 16560 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.899446 16580 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.826659 16500 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:30.930255 13697 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica c243715fe12445019c33bbeafbaea635 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:30.972268 13697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 47465.1us
I20250623 01:59:30.933991 16478 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.935742 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.926056 16558 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.973405 16500 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.975129 16558 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.937353 16560 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:30.937624 16608 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.921548 16484 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.976096 16608 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.977617 16484 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.978076 16608 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.978858 13880 raft_consensus.cc:2391] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 3: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:59:30.938347 16616 log.cc:927] Time spent T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20250623 01:59:30.942210 16578 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:30.944195 16472 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.945070 16527 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.945878 13879 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:30.946272 16540 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:30.950814 16476 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:30.906474 16587 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:30.954993 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:30.999259 13957 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:30.957948 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:30.958568 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:30.959728 13804 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:31.000634 13881 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:31.000864 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.001453 13962 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:30.961328 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.003485 13885 raft_consensus.cc:2391] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:30.962316 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:31.004272 13886 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 5
I20250623 01:59:30.965948 16551 consensus_queue.cc:237] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.970198 16459 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:30.970889 16584 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250623 01:59:30.902380 16598 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:30.980856 13888 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:30.981495 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:31.023185 13954 raft_consensus.cc:2366] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:59:30.982404 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.024284 13887 raft_consensus.cc:2391] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:30.984500 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:30.984805 13812 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250623 01:59:31.025208 13809 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
I20250623 01:59:30.985462 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:30.986474 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:31.026428 13808 raft_consensus.cc:2391] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
W20250623 01:59:30.993600 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 109297us
I20250623 01:59:31.026994 13880 raft_consensus.cc:2391] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:31.027485 14478 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:30.997638 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 125582us
I20250623 01:59:31.007531 13852 leader_election.cc:400] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5.
I20250623 01:59:31.029404 13852 leader_election.cc:403] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:31.030812 16558 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:31.009202 13957 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:31.015846 13886 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:31.008317 13854 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.031327 16558 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.033572 16558 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:31.039901 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 299321us
I20250623 01:59:31.045955 13884 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:31.050091 13888 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:31.051106 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.051719 13954 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:31.052170 16613 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5.
I20250623 01:59:31.049443 16599 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:31.052961 16599 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.055217 13776 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:30.994449 16578 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.056545 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.058883 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.059092 13854 leader_election.cc:400] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:59:31.059542 13854 leader_election.cc:403] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250623 01:59:31.058965 13954 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:31.058177 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.062978 13955 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:31.063436 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.064009 13962 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:31.066790 13854 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:31.067454 13886 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250623 01:59:31.074573 16606 log.cc:927] Time spent T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.244s user 0.001s sys 0.000s
I20250623 01:59:31.076272 16614 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:31.059665 16599 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:31.076790 16418 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 9. Current term is 10.
I20250623 01:59:31.081111 16527 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.078421 16608 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:31.085372 13926 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:31.086050 16540 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.086140 16558 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:31.086616 16558 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.086962 16558 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:31.088128 16605 log.cc:927] Time spent T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.421s user 0.001s sys 0.000s
I20250623 01:59:31.090839 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.092936 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.065515 16518 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:31.093612 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:31.096845 13954 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:31.094053 13955 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:31.094676 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:31.100724 13808 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.098860 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:31.095988 16521 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:31.105065 16454 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:31.107908 13808 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:31.068312 16482 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:59:31.113389 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:31.125224 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.131362 13962 raft_consensus.cc:2391] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:31.126811 13882 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:31.129808 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:31.133903 14515 raft_consensus.cc:2391] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 9: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:31.061125 16492 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.102838 16521 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.055265 16587 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:31.112200 16562 consensus_meta.cc:287] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.508s user 0.011s sys 0.000s
I20250623 01:59:31.141294 13954 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:31.096463 16518 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.145473 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:31.152240 13886 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.147854 13808 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250623 01:59:31.109974 16472 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:31.148612 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.150528 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 118) took 1913 ms. Trace:
I20250623 01:59:31.105597 16454 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.149767 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.154035 13881 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:31.156175 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:31.156986 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.155759 16518 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
W20250623 01:59:31.232548 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 72060.4us
I20250623 01:59:31.240298 13957 raft_consensus.cc:2366] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5.
W20250623 01:59:31.249634 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 181190us
I20250623 01:59:31.138484 16608 raft_consensus.cc:695] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:31.154606 13710 rpcz_store.cc:276] 0623 01:59:29.237461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:29.237580 (+ 119us) service_pool.cc:225] Handling call
0623 01:59:31.150469 (+1912889us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:29.287155 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:29.287620 (+ 465us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:29.287651 (+ 31us) write_op.cc:435] Acquired schema lock
0623 01:59:29.287673 (+ 22us) tablet.cc:660] Decoding operations
0623 01:59:29.287984 (+ 311us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:29.288030 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:29.288064 (+ 34us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:59:29.288247 (+ 183us) tablet.cc:699] Row locks acquired
0623 01:59:29.288267 (+ 20us) write_op.cc:260] PREPARE: finished
0623 01:59:29.288388 (+ 121us) write_op.cc:270] Start()
0623 01:59:29.288483 (+ 95us) write_op.cc:276] Timestamp: P: 1750643969288346 usec, L: 0
0623 01:59:29.288504 (+ 21us) op_driver.cc:348] REPLICATION: starting
0623 01:59:29.289084 (+ 580us) log.cc:844] Serialized 530 byte log entry
0623 01:59:30.221721 (+932637us) op_driver.cc:464] REPLICATION: finished
0623 01:59:30.339769 (+118048us) write_op.cc:301] APPLY: starting
0623 01:59:30.350208 (+ 10439us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:30.367036 (+ 16828us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:30.408120 (+ 41084us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:30.573917 (+165797us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:30.574087 (+ 170us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:59:30.574133 (+ 46us) write_op.cc:312] APPLY: finished
0623 01:59:30.574304 (+ 171us) log.cc:844] Serialized 23 byte log entry
0623 01:59:30.574472 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:30.574565 (+ 93us) write_op.cc:454] Released schema lock
0623 01:59:30.574620 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:30.575024 (+ 404us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:30.575062 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":31880,"apply.run_cpu_time_us":2128,"apply.run_wall_time_us":279610,"mutex_wait_us":165479,"num_ops":1,"prepare.queue_time_us":3203,"prepare.run_cpu_time_us":2170,"prepare.run_wall_time_us":2169,"replication_time_us":921065}]]}
I20250623 01:59:31.150014 16613 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:31.260342 13886 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:31.257139 13953 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:31.256747 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.261744 16525 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:31.265621 16525 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.248274 16418 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:31.251302 16580 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:31.272141 13953 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:31.172828 16562 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.152477 16619 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.118s
I20250623 01:59:31.265842 13962 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:31.288769 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 139328us
I20250623 01:59:31.291684 13852 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.288439 16484 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:31.293283 16484 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.295809 16484 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:31.296922 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.301573 16570 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:31.302856 16570 raft_consensus.cc:695] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:31.303999 16570 consensus_queue.cc:237] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.305541 16478 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:59:31.309553 16608 consensus_queue.cc:237] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.312443 16454 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.314368 13925 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.273702 16514 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:31.323872 13960 raft_consensus.cc:2366] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
W20250623 01:59:31.292236 16579 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.005s sys 0.024s
I20250623 01:59:31.276667 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:31.334244 16603 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:31.334828 16603 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.336916 16603 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:31.341503 16521 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:31.332283 16598 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:31.341866 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 286700us
I20250623 01:59:31.342998 16488 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:31.343492 16488 raft_consensus.cc:695] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:31.325299 16562 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.345921 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:31.346561 13809 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:31.346441 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:31.346743 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.347262 13810 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:31.347376 13808 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:31.349356 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.349903 13961 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:31.353906 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.357192 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.357776 13811 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
W20250623 01:59:31.330278 16579 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.803s user 0.005s sys 0.024s
I20250623 01:59:31.424851 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.421941 13925 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.423553 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:31.264350 16613 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.426474 13879 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:31.344791 16488 consensus_queue.cc:237] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.427275 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.427259 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:31.432826 13804 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.428781 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.284036 16580 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.441954 16580 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.429245 13854 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.431146 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.442488 16620 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:31.443091 16620 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.443215 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.443550 16620 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.437968 13960 raft_consensus.cc:1273] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250623 01:59:31.438032 13809 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:31.445531 16492 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.446897 16609 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:31.447295 16609 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.447645 16609 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:31.447729 16599 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.449653 14511 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 8: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:31.453171 16617 consensus_queue.cc:1035] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:59:31.443897 13808 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:31.456648 13804 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:31.457710 14481 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:31.421209 16601 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.008s sys 0.017s
I20250623 01:59:31.331674 16514 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.458217 13810 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:31.364305 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:31.277726 16418 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.465695 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.473974 13879 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:31.476254 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.481354 13804 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:31.434279 16560 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:31.476874 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.487850 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:31.487998 13809 raft_consensus.cc:2391] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:31.488320 13812 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.477428 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:31.478062 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.489027 13813 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:31.476781 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.489729 13811 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
W20250623 01:59:31.466270 16601 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.008s sys 0.018s
I20250623 01:59:31.493667 13852 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:31.484400 13806 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:31.485180 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:31.496156 13957 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:31.485549 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.482792 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:31.497084 13879 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:31.498776 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:31.498869 13776 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.499454 13886 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.470994 16386 raft_consensus.cc:2747] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:31.467643 16599 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:31.506666 16514 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:31.509882 16462 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:31.510677 13853 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:31.512696 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:31.513239 13806 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 5
I20250623 01:59:31.517638 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.518162 16629 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:31.518640 16629 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.519352 13806 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:31.519356 13879 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:31.524088 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:31.524960 13881 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:31.526086 13925 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.526330 13886 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:31.529599 13926 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:31.530575 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.531250 13806 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:31.532744 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.538234 13812 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
W20250623 01:59:31.548734 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 49975.9us
I20250623 01:59:31.550712 13886 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250623 01:59:31.552976 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 62419.9us
I20250623 01:59:31.503439 16485 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:31.559036 14479 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:31.561707 13925 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.671770 16602 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250623 01:59:31.673604 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 111969us
W20250623 01:59:31.646436 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 83006.8us
I20250623 01:59:31.653141 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37408 (request call id 599) took 1211 ms. Trace:
W20250623 01:59:31.653450 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 97799us
I20250623 01:59:31.544829 16560 raft_consensus.cc:695] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:31.665295 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 106389us
I20250623 01:59:31.567216 16527 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:31.520980 16629 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:31.554337 16617 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:31.560114 16514 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:31.542032 16598 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.099s
I20250623 01:59:31.551753 16462 raft_consensus.cc:695] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:31.557035 16454 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:31.561968 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:31.675974 13958 rpcz_store.cc:276] 0623 01:59:30.430802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:30.430946 (+ 144us) service_pool.cc:225] Handling call
0623 01:59:31.565796 (+1134850us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400073d54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e89f509e 00007f41e209f8c7 00007f41e209f852 00007f41e209f781 00007f41e209f6fc 00007f41e209e9b2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:31.641696 (+ 75900us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":294839168}
I20250623 01:59:31.695307 13806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:34972
I20250623 01:59:31.698140 13779 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:31.702299 13812 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:31.708113 13806 raft_consensus.cc:604] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:31.710420 13779 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:31.696921 16514 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:59:31.718417 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 154428us
I20250623 01:59:31.719065 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.701143 16629 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:31.721202 13806 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:31.699829 16454 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.698791 16527 raft_consensus.cc:695] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:31.731894 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.732720 13957 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:31.722857 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.719388 16601 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:31.734807 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:31.718278 16480 raft_consensus.cc:2762] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20250623 01:59:31.747406 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 72839.9us
I20250623 01:59:31.722443 16514 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.715044 16525 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:31.717725 16617 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20250623 01:59:31.719923 16621 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.044s
I20250623 01:59:31.729688 16454 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:31.736606 16629 raft_consensus.cc:695] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:31.738324 16482 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:31.753540 13806 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250623 01:59:31.743829 16601 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:31.757345 16632 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:31.757828 16632 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.758157 16632 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:31.751166 16609 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.759956 16609 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:31.761334 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.761683 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:31.747695 16418 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:31.763011 13776 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:31.763013 16601 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
W20250623 01:59:31.764737 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 54419.8us
I20250623 01:59:31.790431 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:31.791668 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:31.792369 16575 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:59:31.792874 16575 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:31.793285 16575 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:59:31.767189 16628 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
W20250623 01:59:31.768632 16634 log.cc:927] Time spent T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20250623 01:59:31.776940 16632 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.756331 16482 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.792426 13953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 2292) took 1215 ms. Trace:
W20250623 01:59:31.804708 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 54734.1us
W20250623 01:59:31.767920 16627 log.cc:927] Time spent T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
W20250623 01:59:31.796931 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 51099.6us
I20250623 01:59:31.736044 16560 consensus_queue.cc:237] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.807718 13926 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:31.801887 13779 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:31.779706 13957 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:31.804839 13953 rpcz_store.cc:276] 0623 01:59:30.577368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:30.579929 (+ 2561us) service_pool.cc:225] Handling call
0623 01:59:31.156031 (+576102us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b5800016e98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:59:31.792231 (+636200us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400080a54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:31.792347 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":233706752}
I20250623 01:59:31.794467 16480 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:31.812850 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:31.810197 13926 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:31.816617 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.817205 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:31.752478 16462 consensus_queue.cc:237] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:31.797189 16440 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:31.806131 16482 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.809701 13925 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.788749 16484 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250623 01:59:31.823468 13926 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:31.824193 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:31.826764 13961 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:31.827751 16613 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:31.828506 16633 log.cc:927] Time spent T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20250623 01:59:31.825736 16491 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:31.832067 13779 leader_election.cc:400] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5.
I20250623 01:59:31.832310 13779 leader_election.cc:403] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:31.833235 13779 leader_election.cc:400] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:31.833513 13779 leader_election.cc:403] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:31.821130 16488 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:31.838091 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.835592 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:31.823083 16578 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.839987 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:31.824596 16480 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:31.825551 16440 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:31.836654 16445 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5.
I20250623 01:59:31.845270 16580 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:31.841109 16488 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:31.798820 16527 consensus_queue.cc:237] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.825847 16446 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20250623 01:59:31.839453 16609 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:31.839063 16644 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:31.849850 16440 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:31.862221 13953 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:31.865253 16610 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
W20250623 01:59:31.872267 16558 consensus_meta.cc:287] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.785s user 0.005s sys 0.007s
I20250623 01:59:31.872731 16485 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:31.863507 16488 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:31.865391 16649 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:31.825714 16629 consensus_queue.cc:237] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.883407 13926 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:31.873597 16480 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.875958 16650 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:31.915683 16645 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:31.885615 16644 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:31.869131 16609 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:31.909952 16640 log.cc:927] Time spent T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
I20250623 01:59:31.904464 16649 raft_consensus.cc:695] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:31.879069 16558 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.934180 13954 raft_consensus.cc:2391] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:31.928529 16644 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:31.935189 16644 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:31.929873 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 96761.8us
I20250623 01:59:31.939607 16601 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.882505 16485 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.956038 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:31.932183 16580 raft_consensus.cc:695] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:31.962663 13959 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250623 01:59:31.930853 16632 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.971839 13806 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:31.970276 16620 consensus_meta.cc:287] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.526s user 0.013s sys 0.000s
I20250623 01:59:31.962502 16580 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:31.995416 16578 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:31.996032 16578 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:31.997987 16578 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.958685 16617 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20250623 01:59:32.000305 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:32.003293 13888 raft_consensus.cc:2366] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 7. Current term is 8.
I20250623 01:59:32.003172 13957 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "933266ad75974c449e00b527ab278f8f"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:31.983192 16632 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:32.005896 13776 leader_election.cc:400] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 7. Current term is 8.
I20250623 01:59:32.008894 13776 leader_election.cc:403] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250623 01:59:31.998317 16601 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:31.994673 16620 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.045557 13958 raft_consensus.cc:2391] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:32.046379 16620 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:31.927484 16650 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:32.042311 16518 consensus_meta.cc:287] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.786s user 0.009s sys 0.002s
I20250623 01:59:32.005594 16649 consensus_queue.cc:237] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:32.052800 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 169488us
I20250623 01:59:32.060179 13888 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:59:32.063592 16609 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:32.069587 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:32.070711 16491 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:32.072575 16575 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:32.075124 16624 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.808s user 0.000s sys 0.004s
I20250623 01:59:32.076998 16650 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:32.054795 16518 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.079180 13885 raft_consensus.cc:2391] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:32.081907 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:32.085618 13926 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:32.086586 16665 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20250623 01:59:32.086828 16482 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.087750 16654 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:32.088866 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:32.055652 16558 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:32.079751 16518 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.085477 13806 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:32.092523 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:32.094014 13812 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:32.101677 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:32.113948 13954 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:32.079495 16644 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.135764 13954 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:32.145825 13955 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:32.142207 13812 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:32.095728 16462 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 7. Current term is 8.
I20250623 01:59:32.089293 16482 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.096711 16520 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:32.092614 16480 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20250623 01:59:32.157168 13960 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250623 01:59:32.093616 16514 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.159665 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:32.132737 16485 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:32.089953 16454 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.166996 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:32.169852 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:32.171419 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:32.176613 13888 raft_consensus.cc:2391] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:32.176820 13887 raft_consensus.cc:2391] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:32.177632 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:32.177475 16440 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.179517 16440 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.180392 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:32.181118 13888 raft_consensus.cc:2366] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:32.182228 13776 leader_election.cc:400] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:32.182497 13776 leader_election.cc:403] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250623 01:59:32.182883 13804 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:32.165493 16520 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.189005 13882 raft_consensus.cc:2391] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:32.191995 13804 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:32.193017 16445 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:32.199414 16660 raft_consensus.cc:991] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: : Instructing follower ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:32.200138 16660 raft_consensus.cc:1079] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Signalling peer ee041a8325864e788a383b8356afc652 to start an election
I20250623 01:59:32.214345 13887 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556"
dest_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:58136
I20250623 01:59:32.214931 13887 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:32.215349 13887 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:32.219069 13962 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:32.219573 13957 raft_consensus.cc:604] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:32.219940 13957 raft_consensus.cc:612] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250623 01:59:32.222510 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 184112us
I20250623 01:59:32.261384 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:32.262382 13888 raft_consensus.cc:2366] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4.
W20250623 01:59:32.264598 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 46483.6us
I20250623 01:59:32.268632 13776 leader_election.cc:400] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4.
I20250623 01:59:32.268584 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:32.269513 13776 leader_election.cc:403] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250623 01:59:32.270125 13954 raft_consensus.cc:2391] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:32.193048 16671 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:32.271484 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:32.277846 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:32.279454 14511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48412 (request call id 159) took 1182 ms. Trace:
I20250623 01:59:32.279881 14511 rpcz_store.cc:276] 0623 01:59:31.097084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.098583 (+ 1499us) service_pool.cc:225] Handling call
0623 01:59:32.279264 (+1180681us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b540071ded4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:32.279400 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":798839680}
I20250623 01:59:32.260154 16604 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:32.259948 16660 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:59:32.301856 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 138478us
I20250623 01:59:32.269212 16520 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:32.314031 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
W20250623 01:59:32.364537 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 45245.7us
I20250623 01:59:32.366086 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:32.368134 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:32.369616 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:32.370085 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:32.370739 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:32.371290 16454 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:32.371443 13807 raft_consensus.cc:2391] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:32.372263 16644 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:32.369484 13887 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.373939 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:32.374517 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:32.375777 13926 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:32.306452 16575 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.378221 13887 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.378271 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:32.396796 16670 log.cc:927] Time spent T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
I20250623 01:59:32.398027 16514 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.399431 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:32.290869 16671 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.399988 13960 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:32.303427 16604 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.405543 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:32.407441 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:32.408987 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:32.409579 13813 raft_consensus.cc:3053] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:32.409853 13813 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:32.410101 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:32.411433 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 851812us
I20250623 01:59:32.413627 16520 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:32.416620 13926 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:32.417155 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:32.417748 13926 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:32.418421 13852 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:32.421470 13926 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:32.320644 16641 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.090s
I20250623 01:59:32.410518 13813 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:32.424341 16673 log.cc:927] Time spent T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20250623 01:59:32.424459 16583 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:32.425133 16583 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:32.425469 16671 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:32.314846 16660 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.423137 16601 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:32.428925 16601 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:32.429682 16660 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.448529 13927 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:32.451287 16583 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:32.452015 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:32.452739 13961 raft_consensus.cc:2391] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:32.374790 16657 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20250623 01:59:32.455224 13879 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:32.421728 16466 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:32.456503 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:59:32.447496 16674 log.cc:927] Time spent T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20250623 01:59:32.497471 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:32.497749 16601 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:32.432628 16520 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250623 01:59:32.498790 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 48172.2us
I20250623 01:59:32.498597 13888 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 8: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:32.500788 13959 raft_consensus.cc:2366] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4.
W20250623 01:59:32.501883 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 238184us
I20250623 01:59:32.502300 13854 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:32.503013 16585 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 3. Current term is 4.
I20250623 01:59:32.503057 16518 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:32.504675 13960 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
W20250623 01:59:32.504946 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 50896.5us
I20250623 01:59:32.509230 13854 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:32.505641 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:32.509727 13926 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:32.515199 13960 raft_consensus.cc:1273] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:32.513978 16617 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:32.529773 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:32.531208 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:32.525408 16520 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:32.535610 16570 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:32.536209 16570 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:32.543407 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 56676.3us
I20250623 01:59:32.527817 16418 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
W20250623 01:59:32.524648 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 89394.9us
I20250623 01:59:32.556969 13925 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:32.568120 16488 consensus_meta.cc:287] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.682s user 0.010s sys 0.000s
I20250623 01:59:32.541414 16641 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:32.570401 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
W20250623 01:59:32.611595 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 92031.7us
I20250623 01:59:32.624089 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:32.583598 16580 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:32.554431 16570 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.554911 16604 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:32.554900 16418 raft_consensus.cc:695] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:32.567407 16617 raft_consensus.cc:695] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:59:32.609741 16684 log.cc:927] Time spent T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.170s user 0.002s sys 0.005s
I20250623 01:59:32.650664 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:32.549821 16645 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20250623 01:59:32.574846 16583 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:32.576210 16488 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.656386 16608 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:32.694020 16608 raft_consensus.cc:695] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:32.697216 13885 raft_consensus.cc:1273] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:32.698621 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:32.702759 16687 log.cc:927] Time spent T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.085s user 0.001s sys 0.001s
I20250623 01:59:32.703125 16615 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:32.704556 13884 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250623 01:59:32.705404 13805 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:32.705526 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 177211us
I20250623 01:59:32.708374 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:32.706511 13884 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20250623 01:59:32.632555 16641 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:32.693842 16617 consensus_queue.cc:237] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.720160 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:32.721062 13812 raft_consensus.cc:1273] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:32.696593 16585 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:32.723706 16418 consensus_queue.cc:237] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.704766 16527 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:32.671878 16583 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:32.703691 16615 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:32.719254 16570 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:32.916358 13805 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
W20250623 01:59:32.758222 16680 log.cc:927] Time spent T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250623 01:59:33.074810 13960 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 2.
W20250623 01:59:33.247923 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 487219us
I20250623 01:59:33.250399 13807 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250623 01:59:33.266474 13851 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.026s sys 0.042s
W20250623 01:59:33.266965 13851 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.566s user 0.026s sys 0.042s
I20250623 01:59:33.030341 16691 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:32.757745 16585 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:33.277019 13879 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:33.278437 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:32.763622 16527 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:33.248045 16561 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:33.308756 16690 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.677s user 0.001s sys 0.000s
I20250623 01:59:32.758592 16608 consensus_queue.cc:237] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:33.037127 16570 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:33.267954 16480 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:33.323416 16699 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:33.323642 16527 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:33.324244 16527 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:33.324043 16699 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:33.324739 16527 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:33.326912 16527 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:33.296351 16694 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.677s
I20250623 01:59:33.368134 13708 catalog_manager.cc:5582] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 4, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:33.351574 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:33.481830 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 2311) took 1677 ms. Trace:
I20250623 01:59:33.482345 13958 rpcz_store.cc:276] 0623 01:59:31.803808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.803954 (+ 146us) service_pool.cc:225] Handling call
0623 01:59:33.480500 (+1676546us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b5400395dd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:33.480679 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":519113600}
I20250623 01:59:33.319101 16518 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:33.302770 16691 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:33.450171 16620 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:33.355643 16615 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:33.457652 16583 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:33.550740 13925 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.000s sys 0.000s
W20250623 01:59:33.563351 13925 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.857s user 0.001s sys 0.000s
I20250623 01:59:33.452706 16689 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:33.481458 16488 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:33.524199 16694 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:33.329273 16480 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:33.350122 16699 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:33.587633 16691 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:33.588182 16585 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:33.589164 16570 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:33.616232 16701 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:59:33.624351 13776 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.006s sys 0.013s
W20250623 01:59:33.624749 13776 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.583s user 0.006s sys 0.013s
W20250623 01:59:33.626120 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 865320us
I20250623 01:59:33.626963 13776 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:33.629792 13779 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.011s sys 0.008s
I20250623 01:59:33.557807 16518 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:33.320514 16561 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:33.684509 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 56219.4us
I20250623 01:59:33.612325 16689 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:33.630862 16452 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250623 01:59:33.630182 13779 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.588s user 0.011s sys 0.008s
I20250623 01:59:33.622221 16701 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:33.724973 13927 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.013s sys 0.011s
W20250623 01:59:33.725519 13927 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.693s user 0.013s sys 0.011s
I20250623 01:59:33.580510 16694 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:33.738999 16689 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:33.739570 16689 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:33.740043 16689 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:33.698607 16585 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20250623 01:59:33.698352 16462 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:33.751855 16462 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:33.828117 16462 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:33.729979 16488 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:33.830382 16666 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.012s sys 0.013s
W20250623 01:59:33.868081 13926 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.009s sys 0.016s
W20250623 01:59:33.954126 13926 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.656s user 0.009s sys 0.017s
W20250623 01:59:33.954747 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 676442us
W20250623 01:59:33.853621 13852 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.192s user 0.006s sys 0.019s
W20250623 01:59:33.956055 13852 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.294s user 0.006s sys 0.019s
W20250623 01:59:33.857354 14478 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.202s user 0.014s sys 0.018s
I20250623 01:59:33.945645 16644 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:33.988724 16644 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:33.946345 13925 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 2.081s (SENT) [suppressed 58 similar messages]
I20250623 01:59:33.946828 16452 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:33.947461 16689 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:33.855535 16714 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:33.948882 16620 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:33.950799 16488 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:33.992488 16666 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.878s user 0.012s sys 0.013s
W20250623 01:59:33.957129 14478 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.302s user 0.014s sys 0.018s
I20250623 01:59:33.953446 16561 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:33.990816 13854 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.000s sys 0.031s
W20250623 01:59:33.992251 14481 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.294s user 0.015s sys 0.019s
W20250623 01:59:34.009959 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 53111.6us
W20250623 01:59:34.026522 14481 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.328s user 0.015s sys 0.019s
W20250623 01:59:34.034701 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 1.33658e+06us
W20250623 01:59:34.026508 13854 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.754s user 0.000s sys 0.031s
I20250623 01:59:34.060184 16699 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.017702 16452 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.023749 16620 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:34.124476 13853 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.046s user 0.010s sys 0.021s
W20250623 01:59:34.161048 13853 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.083s user 0.010s sys 0.021s
I20250623 01:59:34.164322 16472 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.157644 16518 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.164849 16472 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.165617 16573 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:34.166193 16518 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:34.166817 16472 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.166723 16518 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.166977 16701 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.168572 16603 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.169090 16603 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.170356 16488 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.171326 16603 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.171841 16480 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.168560 16518 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.172539 16480 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.086570 16705 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:34.109596 16698 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:34.110843 16644 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.173162 16480 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.131933 16615 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:59:34.212994 13887 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.174s user 0.054s sys 0.102s
W20250623 01:59:34.214035 13887 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.175s user 0.054s sys 0.102s
I20250623 01:59:34.166209 16573 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.219821 16452 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.029222 16714 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.223575 16714 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:34.158710 16699 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:34.261148 13955 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 1.592s user 0.026s sys 0.039s
W20250623 01:59:34.261700 13955 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.593s user 0.026s sys 0.039s
W20250623 01:59:34.266402 13883 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.015s sys 0.036s
W20250623 01:59:34.266872 13883 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s user 0.015s sys 0.036s
W20250623 01:59:34.274725 13928 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.242s user 0.014s sys 0.056s
W20250623 01:59:34.275168 13928 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.243s user 0.014s sys 0.056s
I20250623 01:59:34.233490 16691 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.257025 16452 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.215226 16644 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20250623 01:59:34.216241 16615 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.261135 16462 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.288733 16492 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:34.289345 16492 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.268956 16714 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:34.211658 16698 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:34.294320 13778 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.006s sys 0.004s
W20250623 01:59:34.293395 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 563439us
W20250623 01:59:34.302516 13851 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.000s sys 0.030s
W20250623 01:59:34.304570 13851 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.037s user 0.000s sys 0.031s
W20250623 01:59:34.304018 13778 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.558s user 0.006s sys 0.004s
W20250623 01:59:34.298329 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 351972us
W20250623 01:59:34.296326 13888 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.081s user 0.006s sys 0.022s
W20250623 01:59:34.306303 13888 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.091s user 0.006s sys 0.022s
I20250623 01:59:34.206188 16705 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.313688 13888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 2128) took 1615 ms. Trace:
W20250623 01:59:34.310566 16617 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.023s sys 0.070s
I20250623 01:59:34.314366 13888 rpcz_store.cc:276] 0623 01:59:32.691416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.691618 (+ 202us) service_pool.cc:225] Handling call
0623 01:59:32.707319 (+ 15701us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:32.707494 (+ 175us) raft_consensus.cc:1532] Early marking committed up to index 6
0623 01:59:32.725281 (+ 17787us) log.cc:844] Serialized 14 byte log entry
0623 01:59:34.306654 (+1581373us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:34.306712 (+ 58us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:59:34.306789 (+ 77us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:34.306871 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:34.307060 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36322,"prepare.queue_time_us":34382,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":324461,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":16195,"thread_start_us":1580882,"threads_started":2,"wal-append.queue_time_us":602283}
W20250623 01:59:34.318653 13779 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.002s sys 0.011s
W20250623 01:59:34.319092 13779 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.002s sys 0.011s
I20250623 01:59:34.295536 16694 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:59:34.296195 16647 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:34.282881 16452 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:34.314771 16617 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.855s user 0.024s sys 0.070s
I20250623 01:59:34.282274 16691 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.291077 16462 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.323386 16709 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:34.341787 16720 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:34.341919 16713 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.343454 16452 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.349570 16620 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:34.350589 16480 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.311450 16711 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:34.295310 16714 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.325547 16666 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:34.366837 16608 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.831s user 0.004s sys 0.048s
I20250623 01:59:34.361742 16492 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:34.360131 16725 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:34.375041 16480 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:34.504570 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:34.504235 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:34.382511 16608 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s user 0.004s sys 0.048s
I20250623 01:59:34.374344 16713 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.548162 16713 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:34.550967 14478 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.000s sys 0.003s
W20250623 01:59:34.551347 14478 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.529s user 0.000s sys 0.003s
I20250623 01:59:34.551992 16719 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:34.552747 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.552479 16719 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.553117 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.552983 14511 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:34.554795 16719 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:34.555032 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.555969 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.556226 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.556738 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.557026 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.557355 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.557623 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.558537 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.559027 14509 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:34.557659 14512 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:34.561751 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.561956 14509 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:34.562368 14512 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:34.563278 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.353477 16647 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.572309 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 2332) took 1936 ms. Trace:
I20250623 01:59:34.573038 16647 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:34.380281 16711 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.576439 16711 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.573459 13955 rpcz_store.cc:276] 0623 01:59:32.636102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.649642 (+ 13540us) service_pool.cc:225] Handling call
0623 01:59:32.649971 (+ 329us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:32.650203 (+ 232us) raft_consensus.cc:1532] Early marking committed up to index 5
0623 01:59:34.266895 (+1616692us) log.cc:844] Serialized 14 byte log entry
0623 01:59:34.571529 (+304634us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:34.571662 (+ 133us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:59:34.571750 (+ 88us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:34.571935 (+ 185us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:34.572215 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26774,"prepare.queue_time_us":422998,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"thread_start_us":1869791,"threads_started":2}
I20250623 01:59:34.577009 16711 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.578161 16711 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.581578 16711 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.361537 16709 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.590250 16709 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.590952 13808 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:34.526562 16699 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.605262 13778 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:34.607844 16454 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.609447 16466 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.609460 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.562371 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.571087 16617 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.585904 16608 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.585527 16615 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.526010 16480 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.642149 16679 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:34.642624 16679 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.643011 16679 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:34.374689 16720 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.628031 16452 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:34.630174 16709 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.660629 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.654605 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.656451 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.661972 13884 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:59:34.633832 16736 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.648700 16615 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.377784 16666 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.651544 16699 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.682731 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.670522 16707 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:34.634891 16454 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.634724 16466 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.544773 16725 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.649048 16608 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.640354 16617 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.683705 16691 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.674050 16462 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.663928 16452 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.714774 13888 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250623 01:59:34.706595 14509 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20250623 01:59:34.727958 16720 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.729463 13881 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:34.685097 16707 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.731956 13885 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:34.665990 16709 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.734223 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:34.703627 16480 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.738010 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.738540 16725 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.739101 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.666342 16658 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.354s
I20250623 01:59:34.686944 16615 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.702855 16698 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.688045 16699 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.708416 16714 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:34.690013 16573 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.741976 16731 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20250623 01:59:34.744567 16454 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:34.744515 16733 raft_consensus.cc:491] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:34.747956 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:34.756517 16712 log.cc:927] Time spent T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.248s user 0.000s sys 0.002s
I20250623 01:59:34.756765 16733 raft_consensus.cc:513] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.757951 13961 raft_consensus.cc:2408] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20250623 01:59:34.758204 13887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 2129) took 2066 ms. Trace:
I20250623 01:59:34.759076 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:34.755471 16466 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:34.670472 16736 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.748747 16720 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:34.759502 16733 leader_election.cc:290] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.753026 16608 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:34.732112 16462 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.758625 13887 rpcz_store.cc:276] 0623 01:59:32.691963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.692113 (+ 150us) service_pool.cc:225] Handling call
0623 01:59:32.694659 (+ 2546us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:32.762977 (+ 68318us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:32.763006 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:32.763607 (+ 601us) log.cc:844] Serialized 26 byte log entry
0623 01:59:34.214641 (+1451034us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:59:34.214697 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:34.214787 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:34.758017 (+543230us) raft_consensus.cc:1683] finished
0623 01:59:34.758038 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:34.758154 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29435,"raft.queue_time_us":526777,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":20360,"thread_start_us":1238398,"threads_started":2,"wal-append.queue_time_us":1289927}
I20250623 01:59:34.751390 16452 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.768040 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:34.765129 13926 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.019s sys 0.019s
W20250623 01:59:34.769171 13926 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.623s user 0.019s sys 0.019s
W20250623 01:59:34.769039 13697 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20250623 01:59:34.769542 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 645570us
W20250623 01:59:34.770706 14481 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.007s sys 0.025s
W20250623 01:59:34.771054 14481 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.007s sys 0.025s
I20250623 01:59:34.743136 16705 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:34.773700 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:34.778570 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.804579 14509 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:34.804838 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.761098 16714 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.830986 16734 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.829443 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:34.834090 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.837131 13888 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
I20250623 01:59:34.763520 16698 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.763446 16454 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:34.836447 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:34.776448 16608 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.835501 13853 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:34.794827 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.832490 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:34.843174 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1484) took 2466 ms. Trace:
I20250623 01:59:34.844543 13809 rpcz_store.cc:276] 0623 01:59:32.371134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.371240 (+ 106us) service_pool.cc:225] Handling call
0623 01:59:32.371497 (+ 257us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:32.434593 (+ 63096us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:32.434642 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:32.434699 (+ 57us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:59:32.434745 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:34.837738 (+2402993us) spinlock_profiling.cc:243] Waited 62.9 ms on lock 0x7b54001446d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:34.837795 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:34.837939 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":144557440}
I20250623 01:59:34.742511 16726 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:34.848927 16692 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:59:34.847870 16723 log.cc:927] Time spent T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.226s user 0.000s sys 0.002s
I20250623 01:59:34.848122 16677 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:34.863466 13957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:42192 (request call id 12) took 2864 ms. Trace:
I20250623 01:59:34.767045 16720 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.870915 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:34.794724 16718 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:34.876355 14478 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:34.873828 13957 rpcz_store.cc:276] 0623 01:59:31.983246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.983381 (+ 135us) service_pool.cc:225] Handling call
0623 01:59:34.847136 (+2863755us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b5400260554. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a99443 00007f41e89f3c4a 00007f41e209adc4 00007f41e209ad52 00007f41e209ac81 00007f41e209abfc 00007f41e2099eb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:34.847260 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":495970688}
I20250623 01:59:34.794471 16739 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:34.847949 16705 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:34.883198 16648 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:34.872978 13806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 2237) took 3088 ms (client timeout 2844 ms). Trace:
I20250623 01:59:34.781984 16733 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20250623 01:59:34.832192 16714 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.902489 16744 raft_consensus.cc:491] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:34.849412 16726 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.774150 16462 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.903668 16602 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:34.883814 16466 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.849494 16692 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.973917 16730 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:34.831670 16734 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.843720 16698 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.871543 16454 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.850539 16608 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:34.951334 13806 rpcz_store.cc:269] 0623 01:59:31.761272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.761442 (+ 170us) service_pool.cc:225] Handling call
0623 01:59:34.849920 (+3088478us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54004074d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:34.850013 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":481977600}
I20250623 01:59:34.944233 16705 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:34.890969 16666 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.069224 13887 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
W20250623 01:59:35.072548 16724 log.cc:927] Time spent T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.203s user 0.001s sys 0.000s
I20250623 01:59:35.070459 16705 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:35.065903 16709 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.018702 16648 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:34.983943 16466 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.984632 16744 raft_consensus.cc:513] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.969842 16718 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:34.974254 16677 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:34.972491 16739 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.074530 16617 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.253298 16462 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:35.241139 13925 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.036s sys 0.062s
W20250623 01:59:35.284835 13925 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.684s user 0.036s sys 0.062s
I20250623 01:59:35.284961 16744 leader_election.cc:290] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.285578 16744 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.065377 16730 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.286036 16718 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.180482 16741 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.286659 16741 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.252838 16726 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.256907 16454 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.254279 16608 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.108772 16707 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:35.202606 13776 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.017s sys 0.037s
W20250623 01:59:35.293850 13776 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.725s user 0.017s sys 0.040s
W20250623 01:59:35.294459 13779 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.001s sys 0.031s
W20250623 01:59:35.294885 13779 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.002s sys 0.031s
W20250623 01:59:35.202421 13927 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.102s user 0.025s sys 0.040s
W20250623 01:59:35.297125 13927 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.196s user 0.025s sys 0.043s
I20250623 01:59:35.254967 16714 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:35.202697 16649 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 2.989s user 0.041s sys 0.116s
W20250623 01:59:35.241822 16729 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
W20250623 01:59:35.288887 16679 consensus_meta.cc:287] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.645s user 0.010s sys 0.001s
I20250623 01:59:35.330442 16739 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.330801 16755 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.330204 16679 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.326795 16525 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.072582 16733 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:35.256117 16736 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.100167 16699 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:35.328699 16649 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 3.115s user 0.042s sys 0.117s
I20250623 01:59:35.285984 16744 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.333734 16746 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.122282 16731 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:35.331334 16755 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.180116 16692 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:35.408082 13886 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.000s sys 0.041s
W20250623 01:59:35.428993 13886 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s user 0.002s sys 0.042s
W20250623 01:59:35.427835 13926 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.005s sys 0.013s
W20250623 01:59:35.434679 13926 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.006s sys 0.013s
I20250623 01:59:35.439508 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:35.441128 13957 raft_consensus.cc:2366] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:35.338853 16462 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:35.314101 16734 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.407524 16698 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.344892 16709 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.344403 16748 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.413213 16730 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.404991 16705 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.351683 16733 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:35.439308 16484 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.010s sys 0.036s
I20250623 01:59:35.347137 16746 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.343873 16525 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.350867 16731 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.485898 16760 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:35.464280 16679 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.463515 16757 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:35.463362 16677 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.459460 16741 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.479970 16709 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.499774 16755 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.500226 16733 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:35.501603 16749 log.cc:927] Time spent T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.174s user 0.002s sys 0.000s
I20250623 01:59:35.486483 16760 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.483455 16762 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.505198 16762 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.500583 16757 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.506671 16760 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.509614 16762 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:35.536314 13778 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.002s sys 0.014s
W20250623 01:59:35.537233 13778 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.851s user 0.002s sys 0.015s
I20250623 01:59:35.537925 13888 raft_consensus.cc:2366] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 10. Current term is 11.
W20250623 01:59:35.544575 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 99153.8us
I20250623 01:59:35.442039 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.546792 13957 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
W20250623 01:59:35.549568 13854 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.008s sys 0.012s
W20250623 01:59:35.550047 13854 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.963s user 0.008s sys 0.012s
W20250623 01:59:35.550568 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 1.11017e+06us
W20250623 01:59:35.552119 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 229291us
W20250623 01:59:35.435051 14511 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.010s sys 0.010s
I20250623 01:59:35.548822 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:35.573055 16751 log.cc:927] Time spent T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20250623 01:59:35.573215 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:35.573354 13957 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:35.576907 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:35.462513 16648 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.578358 16707 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.578821 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:35.578539 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.579641 16726 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.446599 16462 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.579368 16679 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:35.481997 16484 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s user 0.010s sys 0.038s
I20250623 01:59:35.579334 16707 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.580468 16726 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.580346 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:35.581640 16734 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:35.582980 16466 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.583235 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.564659 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:35.583637 14511 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.855s user 0.010s sys 0.011s
I20250623 01:59:35.573602 14506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:35.569278 16757 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.591794 13853 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:35.503758 16705 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.481209 16748 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.583714 16731 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.598769 16746 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.599120 16525 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.599548 16746 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:35.601096 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:35.603559 14514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:35.603883 14512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:35.503578 16709 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:35.696163 13778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 99580.6us
I20250623 01:59:35.696851 16679 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.697928 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.621301 14513 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:35.642645 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:35.707108 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 586) took 1011 ms. Trace:
I20250623 01:59:35.599043 16741 raft_consensus.cc:491] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:35.723861 16484 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:35.695541 14510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:35.695753 16462 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:35.636153 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 48663.5us
I20250623 01:59:35.694221 16755 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:35.734714 16755 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.737588 16736 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:35.738436 16736 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.737628 16462 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20250623 01:59:35.740788 16736 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.694438 16752 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:35.641660 16734 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.742966 16484 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.742695 14510 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:35.749014 16466 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:35.749409 16734 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.750299 16466 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.752039 14511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48398 (request call id 177) took 1200 ms. Trace:
I20250623 01:59:35.641681 16664 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:35.752494 14511 rpcz_store.cc:276] 0623 01:59:34.552085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:34.552253 (+ 168us) service_pool.cc:225] Handling call
0623 01:59:34.552630 (+ 377us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:34.552852 (+ 222us) raft_consensus.cc:1532] Early marking committed up to index 6
0623 01:59:34.728197 (+175345us) log.cc:844] Serialized 14 byte log entry
0623 01:59:35.749122 (+1020925us) log.cc:844] Serialized 14 byte log entry
0623 01:59:35.751376 (+ 2254us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:35.751504 (+ 128us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:59:35.751613 (+ 109us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:35.751727 (+ 114us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:35.751967 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":652,"raft.run_cpu_time_us":403,"raft.run_wall_time_us":557,"thread_start_us":989095,"threads_started":2,"wal-append.queue_time_us":773153}
I20250623 01:59:35.755085 16709 raft_consensus.cc:491] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
W20250623 01:59:35.755828 13959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 2042) took 3350 ms (client timeout 1738 ms). Trace:
I20250623 01:59:35.755759 16462 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:35.742720 13779 connection.cc:232] RPC call timeout handler was delayed by -1.40256323899996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
I20250623 01:59:35.679281 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41526 (request call id 625) took 2203 ms. Trace:
I20250623 01:59:35.743268 16752 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.769351 16709 raft_consensus.cc:513] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.714150 13955 rpcz_store.cc:276] 0623 01:59:34.695534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:34.695656 (+ 122us) service_pool.cc:225] Handling call
0623 01:59:34.696346 (+ 690us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:34.759512 (+ 63166us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:34.759547 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:34.759588 (+ 41us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:59:34.759614 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:35.706603 (+946989us) spinlock_profiling.cc:243] Waited 63 ms on lock 0x7b54003f7fd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:35.706661 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:35.706897 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":144859392}
I20250623 01:59:35.766867 13807 rpcz_store.cc:276] 0623 01:59:33.475972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:33.480132 (+ 4160us) service_pool.cc:225] Handling call
0623 01:59:33.536458 (+ 56326us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:33.615332 (+ 78874us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:33.615367 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:33.615406 (+ 39us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:59:33.615455 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:35.677992 (+2062537us) spinlock_profiling.cc:243] Waited 78.6 ms on lock 0x7b54001dbd54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:35.678111 (+ 119us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:35.679002 (+ 891us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":180796800}
I20250623 01:59:35.779340 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.780274 13957 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:35.781505 16484 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.781656 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.783072 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.783464 16466 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:35.761979 13959 rpcz_store.cc:269] 0623 01:59:32.404998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.410194 (+ 5196us) service_pool.cc:225] Handling call
0623 01:59:35.755392 (+3345198us) spinlock_profiling.cc:243] Waited 82.8 ms on lock 0x7b540021be54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:35.755636 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":190477568}
I20250623 01:59:35.769573 16462 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.785142 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.786085 13959 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:35.786551 13956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37408 (request call id 602) took 3818 ms. Trace:
I20250623 01:59:35.786598 16462 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.789043 16766 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:35.764906 16679 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:35.778028 13778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 80860.7us
I20250623 01:59:35.787042 13956 rpcz_store.cc:276] 0623 01:59:31.968429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.968578 (+ 149us) service_pool.cc:225] Handling call
0623 01:59:35.764428 (+3795850us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b5400073d54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e89f509e 00007f41e209f8c7 00007f41e209f852 00007f41e209f781 00007f41e209f6fc 00007f41e209e9b2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:35.786327 (+ 21899us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1330552704}
I20250623 01:59:35.786996 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.784665 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:35.796141 13779 connection.cc:232] RPC call timeout handler was delayed by -1.217221640000048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1574s for a response
I20250623 01:59:35.776214 16734 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:35.604629 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:35.762953 16696 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:35.790030 16766 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.783471 16709 leader_election.cc:290] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.802157 14513 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 11.
I20250623 01:59:35.797511 13853 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:35.799899 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.600641 16746 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.809422 13925 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:35.813980 16748 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:35.816618 16734 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:35.820308 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:35.822283 16734 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.818415 16755 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.823372 16491 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:35.818508 16707 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.799353 16696 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.833285 14478 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:35.834733 16696 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.835803 16641 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:35.842438 16669 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 3.654s user 0.015s sys 0.039s
I20250623 01:59:35.837384 13959 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:35.847661 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:35.848408 16675 log.cc:927] Time spent T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 3.425s user 0.017s sys 0.029s
I20250623 01:59:35.848620 16635 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:35.800943 16741 raft_consensus.cc:513] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:35.852157 13953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 574) took 3914 ms. Trace:
W20250623 01:59:35.852676 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 59217.5us
I20250623 01:59:35.854336 16672 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:59:35.854851 16672 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:35.852533 13953 rpcz_store.cc:276] 0623 01:59:31.938049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.938191 (+ 142us) service_pool.cc:225] Handling call
0623 01:59:31.956766 (+ 18575us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:31.957043 (+ 277us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:31.957070 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:31.957534 (+ 464us) log.cc:844] Serialized 26 byte log entry
0623 01:59:32.272606 (+315072us) raft_consensus.cc:1643] Marking committed up to 7
0623 01:59:32.272656 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:32.272737 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:35.843479 (+3570742us) spinlock_profiling.cc:243] Waited 6.63 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:35.851640 (+ 8161us) raft_consensus.cc:1683] finished
0623 01:59:35.851672 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:35.851954 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23877,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"spinlock_wait_cycles":15242624,"thread_start_us":314791,"threads_started":1,"wal-append.queue_time_us":362784}
I20250623 01:59:35.855293 16672 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:35.856012 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:35.857208 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.860122 16752 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.861888 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:35.862201 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:35.862411 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:35.862730 13807 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:35.862824 13886 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:35.863369 13811 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:35.864534 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:35.869875 16472 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:35.871512 13879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 2130) took 3179 ms. Trace:
I20250623 01:59:35.871935 13879 rpcz_store.cc:276] 0623 01:59:32.692425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.692554 (+ 129us) service_pool.cc:225] Handling call
0623 01:59:32.692910 (+ 356us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:33.277511 (+584601us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:35.871041 (+2593530us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b540018ea54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:35.871335 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1342868736}
I20250623 01:59:35.871096 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:35.874886 13805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1488) took 3502 ms (client timeout 1877 ms). Trace:
W20250623 01:59:35.875399 13805 rpcz_store.cc:269] 0623 01:59:32.372690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.373739 (+ 1049us) service_pool.cc:225] Handling call
0623 01:59:35.874475 (+3500736us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b540016dad4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:35.874658 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":760808704}
I20250623 01:59:35.876871 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:35.864540 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 2257) took 1360 ms. Trace:
I20250623 01:59:35.877737 13812 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:35.872465 14478 leader_election.cc:400] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 pre-election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 10. Current term is 11.
I20250623 01:59:35.877460 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 1569) took 1223 ms. Trace:
I20250623 01:59:35.878927 14478 leader_election.cc:403] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
W20250623 01:59:35.879134 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 46960.9us
I20250623 01:59:35.877892 13808 rpcz_store.cc:276] 0623 01:59:34.504230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:34.504355 (+ 125us) service_pool.cc:225] Handling call
0623 01:59:35.864090 (+1359735us) spinlock_profiling.cc:243] Waited 80.2 ms on lock 0x7b54001fa4d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:35.864343 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":184358784}
I20250623 01:59:35.879639 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:35.878693 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:35.879009 13884 rpcz_store.cc:276] 0623 01:59:34.653829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:34.664995 (+ 11166us) service_pool.cc:225] Handling call
0623 01:59:34.665257 (+ 262us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:34.764223 (+ 98966us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:34.764250 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:34.764290 (+ 40us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:59:34.764319 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:35.876883 (+1112564us) spinlock_profiling.cc:243] Waited 98.8 ms on lock 0x7b540029b3d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:35.876944 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:35.877235 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":227197312}
I20250623 01:59:35.890417 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 256) took 6828 ms. Trace:
I20250623 01:59:35.890760 16766 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:35.893718 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:35.894639 13879 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:35.896392 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:35.892447 13962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 2313) took 4081 ms (client timeout 2844 ms). Trace:
W20250623 01:59:35.890925 13960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 2333) took 3241 ms (client timeout 1803 ms). Trace:
I20250623 01:59:35.881278 16741 leader_election.cc:290] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:35.897624 13962 rpcz_store.cc:269] 0623 01:59:31.811179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.812561 (+ 1382us) service_pool.cc:225] Handling call
0623 01:59:35.892012 (+4079451us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400260554. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:35.892240 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":919241472}
W20250623 01:59:35.937528 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 201416us
I20250623 01:59:36.822356 13853 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:35.911505 13960 rpcz_store.cc:269] 0623 01:59:32.649579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.650392 (+ 813us) service_pool.cc:225] Handling call
0623 01:59:35.890580 (+3240188us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b54000bbdd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:35.890733 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":957804928}
I20250623 01:59:36.814885 16746 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:35.938161 13888 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
W20250623 01:59:36.824445 13888 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 2148) took 1790 ms (client timeout 2004 ms). Trace:
W20250623 01:59:36.824930 13888 rpcz_store.cc:269] 0623 01:59:35.034103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.874494 (+840391us) service_pool.cc:225] Handling call
0623 01:59:36.824383 (+949889us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.082654 13954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 2323) took 3732 ms. Trace:
I20250623 01:59:36.826066 13954 rpcz_store.cc:276] 0623 01:59:32.317066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.317173 (+ 107us) service_pool.cc:225] Handling call
0623 01:59:32.317512 (+ 339us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:32.318243 (+ 731us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:32.318280 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:32.362707 (+ 44427us) log.cc:844] Serialized 26 byte log entry
0623 01:59:32.430673 (+ 67966us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:59:32.430729 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:32.430787 (+ 58us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:35.848972 (+3418185us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b240018a088. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0997bc4 00007f41e0995ccb 00007f41e0a24a89 00007f41e0a24715 00007f41e0a23c70 00007f41e370b228 00007f41e370b988 00007f41e36e31c8 00007f41e36de5aa 00007f41e36dd9f6
0623 01:59:36.043138 (+194166us) raft_consensus.cc:1683] finished
0623 01:59:36.043168 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:36.049172 (+ 6004us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27970,"raft.queue_time_us":841,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":53400,"spinlock_wait_cycles":38274816,"thread_start_us":25485,"threads_started":1,"wal-append.queue_time_us":61946}
I20250623 01:59:35.933200 16580 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.943s
I20250623 01:59:36.162778 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:36.829711 16483 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250623 01:59:36.830381 14507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48412 (request call id 166) took 1284 ms. Trace:
I20250623 01:59:36.830754 14507 rpcz_store.cc:276] 0623 01:59:35.546433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.569039 (+ 22606us) service_pool.cc:225] Handling call
0623 01:59:36.162348 (+593309us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b58001e2a98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:59:36.830298 (+667950us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8667520}
W20250623 01:59:36.381347 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 148487us
I20250623 01:59:35.878487 13708 catalog_manager.cc:5582] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:36.834216 13708 catalog_manager.cc:5582] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 7 to 8, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 8 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:36.334882 16649 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:35.890908 13711 rpcz_store.cc:276] 0623 01:59:29.061816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:29.062913 (+ 1097us) service_pool.cc:225] Handling call
0623 01:59:35.890157 (+6827244us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:29.244776 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:29.285053 (+ 40277us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:29.285085 (+ 32us) write_op.cc:435] Acquired schema lock
0623 01:59:29.285113 (+ 28us) tablet.cc:660] Decoding operations
0623 01:59:29.285515 (+ 402us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:29.285560 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:29.285583 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0623 01:59:29.285905 (+ 322us) tablet.cc:699] Row locks acquired
0623 01:59:29.285921 (+ 16us) write_op.cc:260] PREPARE: finished
0623 01:59:29.286043 (+ 122us) write_op.cc:270] Start()
0623 01:59:29.286172 (+ 129us) write_op.cc:276] Timestamp: P: 1750643969286001 usec, L: 0
0623 01:59:29.286192 (+ 20us) op_driver.cc:348] REPLICATION: starting
0623 01:59:29.286757 (+ 565us) log.cc:844] Serialized 913 byte log entry
0623 01:59:29.945206 (+658449us) op_driver.cc:464] REPLICATION: finished
0623 01:59:30.188608 (+243402us) write_op.cc:301] APPLY: starting
0623 01:59:30.198135 (+ 9527us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:30.221536 (+ 23401us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:30.295370 (+ 73834us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:30.572708 (+277338us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:30.572886 (+ 178us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 01:59:30.572928 (+ 42us) write_op.cc:312] APPLY: finished
0623 01:59:30.573126 (+ 198us) log.cc:844] Serialized 29 byte log entry
0623 01:59:30.573296 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:30.599087 (+ 25791us) write_op.cc:454] Released schema lock
0623 01:59:30.599128 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:30.599438 (+ 310us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:30.599467 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":556329,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":231514,"apply.run_cpu_time_us":5644,"apply.run_wall_time_us":413229,"num_ops":2,"prepare.queue_time_us":114498,"prepare.run_cpu_time_us":2631,"prepare.run_wall_time_us":46161,"replication_time_us":653240,"thread_start_us":213417,"threads_started":1}]]}
I20250623 01:59:36.007726 13805 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
I20250623 01:59:36.011355 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:35.941892 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1565) took 1251 ms. Trace:
I20250623 01:59:35.938182 13880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 2115) took 4055 ms. Trace:
I20250623 01:59:35.938329 16688 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 10. Current term is 11.
I20250623 01:59:36.850548 13881 rpcz_store.cc:276] 0623 01:59:34.652596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:34.653044 (+ 448us) service_pool.cc:225] Handling call
0623 01:59:35.882776 (+1229732us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b540023c8d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:35.903947 (+ 21171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":153935872}
I20250623 01:59:36.866191 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:36.869755 13882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1540) took 5031 ms (client timeout 2081 ms). Trace:
I20250623 01:59:36.871088 13805 raft_consensus.cc:2466] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
W20250623 01:59:36.870283 13882 rpcz_store.cc:269] 0623 01:59:31.837700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.837848 (+ 148us) service_pool.cc:225] Handling call
0623 01:59:36.869207 (+5031359us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400280ad4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:36.869540 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":803578240}
I20250623 01:59:36.872196 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.873118 13805 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:36.873440 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.874449 13807 raft_consensus.cc:2391] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 6: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:36.872057 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.875362 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.876243 13804 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:36.877405 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.878199 13804 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:36.879407 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.875313 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.870316 16612 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:36.880818 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.881472 13807 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:36.881613 13708 catalog_manager.cc:5582] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:36.882633 13884 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:36.883358 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.873150 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:36.882339 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:36.883437 13884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1579) took 1561 ms (client timeout 1801 ms). Trace:
I20250623 01:59:36.884428 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1577) took 1565 ms. Trace:
W20250623 01:59:36.884344 13884 rpcz_store.cc:269] 0623 01:59:35.322270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.950739 (+628469us) service_pool.cc:225] Handling call
0623 01:59:36.883350 (+932611us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.884799 13882 rpcz_store.cc:276] 0623 01:59:35.319241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.872849 (+1553608us) service_pool.cc:225] Handling call
0623 01:59:36.884376 (+ 11527us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.885435 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:36.886018 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:36.886158 13884 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:36.886647 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1583) took 1558 ms. Trace:
I20250623 01:59:36.886687 13882 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:36.887033 13884 rpcz_store.cc:276] 0623 01:59:35.328026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.885081 (+1557055us) service_pool.cc:225] Handling call
0623 01:59:36.886595 (+ 1514us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.874318 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.887924 13805 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:36.888067 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:36.888818 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.851999 13880 rpcz_store.cc:276] 0623 01:59:31.882570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.882673 (+ 103us) service_pool.cc:225] Handling call
0623 01:59:31.883881 (+ 1208us) raft_consensus.cc:1469] Updating replica for 2 ops
0623 01:59:31.884368 (+ 487us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:31.884391 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0623 01:59:31.918756 (+ 34365us) log.cc:844] Serialized 574 byte log entry
0623 01:59:32.056189 (+137433us) raft_consensus.cc:1643] Marking committed up to 7
0623 01:59:32.056245 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:32.056321 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:35.841149 (+3784828us) spinlock_profiling.cc:243] Waited 9.84 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:35.937701 (+ 96552us) raft_consensus.cc:1683] finished
0623 01:59:35.937725 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:35.938014 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":28037,"spinlock_wait_cycles":22627200,"thread_start_us":104640,"threads_started":1,"wal-append.queue_time_us":213621}
I20250623 01:59:36.890205 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.891396 13807 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 4.
I20250623 01:59:36.887235 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1582) took 1561 ms. Trace:
I20250623 01:59:36.893713 13882 rpcz_store.cc:276] 0623 01:59:35.325649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.885649 (+1560000us) service_pool.cc:225] Handling call
0623 01:59:36.887189 (+ 1540us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.883584 13708 catalog_manager.cc:5582] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 3 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:36.897045 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.897795 13708 catalog_manager.cc:5582] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 4 to 6, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 6 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:36.898762 13804 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:36.896127 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 615) took 1358 ms. Trace:
I20250623 01:59:36.899333 13881 rpcz_store.cc:276] 0623 01:59:35.537970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.894800 (+1356830us) service_pool.cc:225] Handling call
0623 01:59:36.895303 (+ 503us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:36.895703 (+ 400us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:36.895741 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:36.895795 (+ 54us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:59:36.895852 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:36.895916 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:36.896032 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.901096 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:36.901376 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 616) took 1362 ms. Trace:
I20250623 01:59:36.901847 13962 raft_consensus.cc:2466] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:36.901801 13881 rpcz_store.cc:276] 0623 01:59:35.538695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.900402 (+1361707us) service_pool.cc:225] Handling call
0623 01:59:36.900752 (+ 350us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:36.901009 (+ 257us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:36.901037 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:36.901093 (+ 56us) raft_consensus.cc:1643] Marking committed up to 7
0623 01:59:36.901122 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:36.901174 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:36.901273 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250623 01:59:36.903216 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 45318.7us
I20250623 01:59:36.903712 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 617) took 1350 ms. Trace:
I20250623 01:59:36.904170 13881 rpcz_store.cc:276] 0623 01:59:35.553759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.902850 (+1349091us) service_pool.cc:225] Handling call
0623 01:59:36.903117 (+ 267us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:36.903388 (+ 271us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:36.903411 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:36.903466 (+ 55us) raft_consensus.cc:1643] Marking committed up to 7
0623 01:59:36.903511 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:36.903566 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:36.903661 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.898116 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.905799 13807 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:36.909476 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.909763 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:36.907495 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 618) took 1353 ms. Trace:
I20250623 01:59:36.890080 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.910710 13954 raft_consensus.cc:2466] T de00fb2e24074586bfe02d33fb92e4e6 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:36.911664 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.912925 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:36.913609 13805 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 3.
I20250623 01:59:36.892690 13884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 2144) took 1859 ms. Trace:
I20250623 01:59:36.915254 13884 rpcz_store.cc:276] 0623 01:59:35.032938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.889049 (+1856111us) service_pool.cc:225] Handling call
0623 01:59:36.892047 (+ 2998us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:36.892366 (+ 319us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:36.892394 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:36.892449 (+ 55us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:59:36.892484 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:36.892535 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:36.892627 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.916570 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:36.908466 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:36.917631 13886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1580) took 1594 ms (client timeout 1880 ms). Trace:
I20250623 01:59:36.917809 13804 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
W20250623 01:59:36.918170 13886 rpcz_store.cc:269] 0623 01:59:35.323082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.858776 (+1535694us) service_pool.cc:225] Handling call
0623 01:59:36.917549 (+ 58773us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24448}
I20250623 01:59:36.919452 13804 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:36.910230 13807 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:36.921010 13881 rpcz_store.cc:276] 0623 01:59:35.554247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.906337 (+1352090us) service_pool.cc:225] Handling call
0623 01:59:36.906772 (+ 435us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:36.907127 (+ 355us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:36.907157 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:36.907213 (+ 56us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:59:36.907289 (+ 76us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:36.907342 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:36.907448 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:36.917045 13805 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250623 01:59:36.928489 16672 consensus_meta.cc:287] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 1.071s user 0.015s sys 0.000s
I20250623 01:59:36.929067 13804 raft_consensus.cc:1273] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:36.936332 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:36.970646 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 76208.1us
I20250623 01:59:36.974118 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:36.952870 16672 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:36.981552 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 108248us
I20250623 01:59:36.977972 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:36.983809 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 1581) took 1659 ms. Trace:
I20250623 01:59:36.984567 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:36.985966 14479 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:36.987120 13806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41526 (request call id 626) took 1111 ms. Trace:
I20250623 01:59:36.987493 13806 rpcz_store.cc:276] 0623 01:59:35.875404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.879456 (+ 4052us) service_pool.cc:225] Handling call
0623 01:59:36.986283 (+1106827us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b0323e 0000000000356d09 00007f41e897bdc0 00007f41e8a427d4 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:59:36.986433 (+ 150us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:36.986658 (+ 225us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:36.986687 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:36.986745 (+ 58us) raft_consensus.cc:1643] Marking committed up to 8
0623 01:59:36.986774 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:36.986831 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:36.986940 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13344640}
I20250623 01:59:36.957098 16649 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:36.984148 13882 rpcz_store.cc:276] 0623 01:59:35.324230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.894459 (+1570229us) service_pool.cc:225] Handling call
0623 01:59:36.894917 (+ 458us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:36.895439 (+ 522us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:36.895469 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:36.895976 (+ 507us) log.cc:844] Serialized 26 byte log entry
0623 01:59:36.981664 (+ 85688us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:59:36.981707 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:36.981773 (+ 66us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:36.983581 (+ 1808us) raft_consensus.cc:1683] finished
0623 01:59:36.983600 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:36.983724 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1059,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":134,"thread_start_us":85259,"threads_started":1,"wal-append.queue_time_us":85635}
I20250623 01:59:37.001791 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:37.002701 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.003010 13804 raft_consensus.cc:2374] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8. Re-sending same reply.
I20250623 01:59:37.004158 13953 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:37.005460 16711 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:37.005679 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.008970 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.009747 13956 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:37.010629 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.011636 13805 raft_consensus.cc:1273] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:37.010723 13883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 599) took 5667 ms. Trace:
I20250623 01:59:37.012651 16762 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:37.012861 16615 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.013579 16615 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:37.013460 16762 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:37.008560 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.011858 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.015461 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.012180 13883 rpcz_store.cc:276] 0623 01:59:31.343196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.343405 (+ 209us) service_pool.cc:225] Handling call
0623 01:59:31.343692 (+ 287us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:31.478584 (+134892us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:31.478605 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:32.007751 (+529146us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b48001b0258. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e8956871 00007f41e894da17 00007f41e894d8aa 00007f41e8a4297f 00007f41e8a42647 00007f41e8a56c6c 00007f41e8a56c18 00007f41e8a56b98 00007f41e8a56b53 00007f41e8a55dfc 00007f41e79dc736
0623 01:59:33.579769 (+1572018us) spinlock_profiling.cc:243] Waited 3.13 ms on lock 0x7b2c00010da8. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e24426d1 00007f41e2451d02 00007f41e3adac51 00007f41e3a9f02a 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832
0623 01:59:33.612685 (+ 32916us) log.cc:844] Serialized 26 byte log entry
0623 01:59:34.267395 (+654710us) raft_consensus.cc:1643] Marking committed up to 7
0623 01:59:34.267436 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:37.010233 (+2742797us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b540018ef54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:37.010299 (+ 66us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:37.010371 (+ 72us) raft_consensus.cc:1683] finished
0623 01:59:37.010384 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:37.010551 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7246,"raft.queue_time_us":7630,"raft.run_cpu_time_us":3306,"raft.run_wall_time_us":526910,"spinlock_wait_cycles":295725824,"thread_start_us":654334,"threads_started":1,"wal-append.queue_time_us":1132339,"wal-append.run_cpu_time_us":4154,"wal-append.run_wall_time_us":1379430}
I20250623 01:59:37.015976 16781 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:37.019352 16781 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:37.017628 13954 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:37.021153 16781 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:37.023815 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37424 (request call id 2050) took 4309 ms. Trace:
I20250623 01:59:37.024804 13958 rpcz_store.cc:276] 0623 01:59:32.707394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:33.543889 (+836495us) service_pool.cc:225] Handling call
0623 01:59:33.578716 (+ 34827us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:33.584680 (+ 5964us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:33.584725 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:33.585262 (+ 537us) log.cc:844] Serialized 26 byte log entry
0623 01:59:35.761534 (+2176272us) spinlock_profiling.cc:243] Waited 8.68 ms on lock 0x7b1800458fe0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b22bc5 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e36dd04b 00007f41e36e2b53 00007f41e36e2f01 00007f41e3a71ccf
0623 01:59:36.128194 (+366660us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:59:36.168942 (+ 40748us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:36.259391 (+ 90449us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:37.014392 (+755001us) raft_consensus.cc:1683] finished
0623 01:59:37.014411 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:37.017178 (+ 2767us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6828,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":18316,"spinlock_wait_cycles":19955584,"thread_start_us":119298,"threads_started":1,"wal-append.queue_time_us":3340395}
I20250623 01:59:37.026615 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.026827 16711 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:37.027374 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37424 (request call id 2069) took 1271 ms. Trace:
I20250623 01:59:37.027735 13955 rpcz_store.cc:276] 0623 01:59:35.756116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:37.026340 (+1270224us) service_pool.cc:225] Handling call
0623 01:59:37.027320 (+ 980us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:37.028861 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:37.028237 13885 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1573) took 2373 ms (client timeout 1553 ms). Trace:
I20250623 01:59:37.030555 13957 raft_consensus.cc:2466] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
W20250623 01:59:37.031510 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 85312.2us
W20250623 01:59:37.029548 13885 rpcz_store.cc:269] 0623 01:59:34.654982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:34.656154 (+ 1172us) service_pool.cc:225] Handling call
0623 01:59:37.027913 (+2371759us) spinlock_profiling.cc:243] Waited 70.6 ms on lock 0x7b540008f054. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:37.028046 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162384128}
I20250623 01:59:37.039346 16672 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:37.031031 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.040716 13810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 2265) took 1042 ms. Trace:
I20250623 01:59:37.014196 16615 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:37.041558 16711 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.041095 13810 rpcz_store.cc:276] 0623 01:59:35.998634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.049644 (+ 51010us) service_pool.cc:225] Handling call
0623 01:59:37.030187 (+980543us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:59:37.040647 (+ 10460us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53930752}
I20250623 01:59:37.058943 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:36.853080 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 424621us
I20250623 01:59:37.063306 16615 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.065510 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.066469 13957 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
W20250623 01:59:37.067641 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 206249us
I20250623 01:59:37.048413 16762 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:37.086360 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 83138.4us
I20250623 01:59:37.091516 13925 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:37.093739 16762 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:37.094297 16762 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:37.094759 16762 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:37.093674 14479 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:37.085654 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.098750 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.110191 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.112151 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.112877 13888 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
W20250623 01:59:37.113413 13888 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 2146) took 2079 ms (client timeout 2353 ms). Trace:
W20250623 01:59:37.113765 13888 rpcz_store.cc:269] 0623 01:59:35.033544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:37.111821 (+2078277us) service_pool.cc:225] Handling call
0623 01:59:37.113358 (+ 1537us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:37.118134 16781 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:37.120077 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:37.120517 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 298312us
I20250623 01:59:37.120826 14508 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
W20250623 01:59:37.121526 13887 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59310 (request call id 610) took 2355 ms (client timeout 1826 ms). Trace:
I20250623 01:59:37.122006 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.121462 13853 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:37.121455 14508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48398 (request call id 189) took 1572 ms. Trace:
I20250623 01:59:37.125988 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.130712 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:37.132362 16581 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:37.125253 14508 rpcz_store.cc:276] 0623 01:59:35.549240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.568111 (+ 18871us) service_pool.cc:225] Handling call
0623 01:59:37.119795 (+1551684us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b58001e2a98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b031ec 00007f41e0b02dab 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:59:37.121381 (+ 1586us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7682688}
W20250623 01:59:37.122020 13887 rpcz_store.cc:269] 0623 01:59:34.766368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:34.766582 (+ 214us) service_pool.cc:225] Handling call
0623 01:59:37.121191 (+2354609us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b5400127854. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:37.121345 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":691105536}
I20250623 01:59:37.133821 16581 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:37.135807 13880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 2145) took 2100 ms. Trace:
I20250623 01:59:37.136300 13880 rpcz_store.cc:276] 0623 01:59:35.033245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:36.890910 (+1857665us) service_pool.cc:225] Handling call
0623 01:59:36.891630 (+ 720us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:36.892467 (+ 837us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:36.892513 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:36.929693 (+ 37180us) log.cc:844] Serialized 26 byte log entry
0623 01:59:37.111217 (+181524us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:59:37.111257 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:37.111329 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:37.133191 (+ 21862us) raft_consensus.cc:1683] finished
0623 01:59:37.133206 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:37.133357 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":175,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":136301,"threads_started":1,"wal-append.queue_time_us":169363}
I20250623 01:59:37.138760 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.117175 16720 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.141111 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.141712 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.141928 13960 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:37.142594 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.143091 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.143294 16780 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:37.133410 16781 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.152539 14508 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20250623 01:59:37.153550 16793 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:37.153909 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:37.154734 14507 raft_consensus.cc:2391] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 8: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:37.156738 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.157891 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.158938 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.159610 13955 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:37.160427 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.144616 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.162950 13957 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:37.144800 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.144040 16601 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:37.145535 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.146001 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.136883 16581 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:37.151620 13954 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:37.165783 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1497) took 1285 ms. Trace:
W20250623 01:59:37.166394 13854 connection.cc:232] RPC call timeout handler was delayed by -1.3539561279999361s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1787s for a response
I20250623 01:59:37.178759 13808 rpcz_store.cc:276] 0623 01:59:35.883492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:37.135577 (+1252085us) service_pool.cc:225] Handling call
0623 01:59:37.165705 (+ 30128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:37.170531 13810 raft_consensus.cc:1238] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:37.170619 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:37.171020 13958 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:37.171463 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:37.172000 16780 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:37.192875 16780 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.175743 13956 raft_consensus.cc:2391] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 3: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:37.173741 16744 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:37.177403 13959 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:37.173748 13853 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:37.176947 13953 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250623 01:59:37.179210 13854 connection.cc:232] RPC call timeout handler was delayed by -1.3119002820000105s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17600000000000002s for a response
I20250623 01:59:37.186350 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.187029 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.182627 13813 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
W20250623 01:59:37.208848 13854 connection.cc:232] RPC call timeout handler was delayed by -1.1484453040000062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
I20250623 01:59:37.181735 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:37.209246 13854 connection.cc:232] RPC call timeout handler was delayed by -1.0710453039999948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2195s for a response
I20250623 01:59:37.194456 16609 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:37.210290 16609 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:37.210778 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:37.211638 13958 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:37.212653 16609 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.216271 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.216794 16478 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:37.216974 13882 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:37.217213 16649 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:37.190246 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.217445 16478 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:37.218305 13804 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:37.201761 14513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:37.201959 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.219372 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.203143 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.220242 16478 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:37.219377 14513 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:37.220760 13954 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 10.
I20250623 01:59:37.221139 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.221464 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.221858 13883 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:37.200199 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 2261) took 1339 ms. Trace:
I20250623 01:59:37.222425 13809 rpcz_store.cc:276] 0623 01:59:35.861159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.861440 (+ 281us) service_pool.cc:225] Handling call
0623 01:59:37.199379 (+1337939us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:59:37.199596 (+ 217us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:37.199773 (+ 177us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:37.199800 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:37.199858 (+ 58us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:59:37.199892 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:37.199942 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:37.200036 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7147776}
I20250623 01:59:37.204767 14511 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:37.224009 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:37.203681 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.225980 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.226436 16609 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election won for term 11
W20250623 01:59:37.219980 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 2264) took 1322 ms (client timeout 1760 ms). Trace:
I20250623 01:59:37.227097 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:37.227236 13812 rpcz_store.cc:269] 0623 01:59:35.932095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.997775 (+ 65680us) service_pool.cc:225] Handling call
0623 01:59:37.201802 (+1204027us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b5800000f98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:59:37.219914 (+ 18112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5984128}
I20250623 01:59:37.228554 16609 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:37.204893 16658 consensus_queue.cc:1046] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250623 01:59:37.230168 16609 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:37.232255 16658 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:37.232658 16658 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:37.233528 16658 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:37.225281 14508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:37.220096 13882 raft_consensus.cc:2466] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:37.237488 13955 raft_consensus.cc:1273] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:37.240468 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.241346 13808 raft_consensus.cc:2466] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 2.
I20250623 01:59:37.204813 16726 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.248832 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.264814 13879 raft_consensus.cc:2391] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:59:37.265467 13879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58122 (request call id 1576) took 1965 ms (client timeout 2081 ms). Trace:
I20250623 01:59:37.261866 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.266024 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.266333 13804 raft_consensus.cc:2466] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:37.267221 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.267371 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.267820 13805 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:37.268668 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.270025 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.270716 13810 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:37.271512 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.272023 13810 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:37.272727 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.274009 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.274602 13809 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:37.275507 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.275828 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.276216 13883 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
W20250623 01:59:37.265823 13879 rpcz_store.cc:269] 0623 01:59:35.299642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:37.248411 (+1948769us) service_pool.cc:225] Handling call
0623 01:59:37.265384 (+ 16973us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 01:59:37.277761 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.268926 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.276474 13881 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 10.
I20250623 01:59:37.284973 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.250830 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.287593 13813 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:37.289805 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.290285 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:37.295989 13813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48572 (request call id 2250) took 4907 ms (client timeout 1740 ms). Trace:
I20250623 01:59:37.296656 13880 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:37.296090 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:37.296916 13813 rpcz_store.cc:269] 0623 01:59:32.381418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.406199 (+ 24781us) service_pool.cc:225] Handling call
0623 01:59:37.182140 (+4775941us) spinlock_profiling.cc:243] Waited 4.6 s on lock 0x7b58008969b0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a19e73 00007f41e3a7d1a7 00007f41e3a95bd7 00007f41e3a973ce 00007f41e3aa2e5c 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc
0623 01:59:37.288395 (+106255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10573851008}
I20250623 01:59:37.307174 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.250218 16658 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:37.315688 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.318601 13954 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:37.327291 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:37.337702 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.314980 16575 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:37.341224 14515 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48398 (request call id 190) took 1791 ms (client timeout 2183 ms). Trace:
I20250623 01:59:37.341620 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:37.350729 14515 rpcz_store.cc:269] 0623 01:59:35.549865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.569104 (+ 19239us) service_pool.cc:225] Handling call
0623 01:59:37.300055 (+1730951us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b58001e2a98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b0323e 0000000000356d09 00007f41e897bdc0 00007f41e8a427d4 00007f41e8a42841 00007f41e89f15e0 00007f41e89f0ec4 00007f41e208ccc4 00007f41e208cc52
0623 01:59:37.341153 (+ 41098us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6049664}
W20250623 01:59:37.390702 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 50282.6us
I20250623 01:59:37.407100 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:37.394673 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 54475.6us
I20250623 01:59:37.406517 13955 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:37.420714 13888 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:37.424199 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.425046 13960 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
W20250623 01:59:37.498544 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 68317.9us
I20250623 01:59:37.500247 16581 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:37.511042 13925 leader_election.cc:336] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): Timed out: RequestConsensusVote RPC to 127.12.87.2:43055 timed out after 2.081s (SENT)
W20250623 01:59:37.511762 16796 log.cc:927] Time spent T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20250623 01:59:37.533288 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.534962 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.535769 13958 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:37.537205 16629 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:37.537869 16629 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:37.539173 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.540196 16629 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:37.540234 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.390429 16575 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:37.547288 13811 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20250623 01:59:37.553735 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 460145us
I20250623 01:59:37.554092 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.554582 14479 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:37.555488 16762 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:37.557826 16805 log.cc:927] Time spent T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.216s user 0.001s sys 0.000s
I20250623 01:59:37.558027 16762 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:37.558847 16799 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:37.559317 16802 log.cc:927] Time spent T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250623 01:59:37.561228 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.566738 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.570449 16575 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.573069 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 2263) took 1708 ms. Trace:
I20250623 01:59:37.573590 13811 rpcz_store.cc:276] 0623 01:59:35.864424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:35.867025 (+ 2601us) service_pool.cc:225] Handling call
0623 01:59:37.295703 (+1428678us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:37.547699 (+251996us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:37.571761 (+ 24062us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b5400366fd4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:37.572844 (+ 1083us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400366fd0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9e49d 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:37.572992 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3845415424}
W20250623 01:59:37.606702 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 212239us
W20250623 01:59:37.608810 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 45351.3us
I20250623 01:59:37.612406 14478 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:37.614106 14478 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:37.614282 16581 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.614929 16574 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:37.615443 16574 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:37.615892 16574 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:37.616025 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.617092 13888 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
W20250623 01:59:37.617656 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 60699.4us
I20250623 01:59:37.617956 13887 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
W20250623 01:59:37.618786 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:37.624312 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.627846 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:37.628763 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 117784us
I20250623 01:59:37.629050 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:37.637082 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:37.637509 13883 raft_consensus.cc:1273] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:59:37.638909 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.640025 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:37.640339 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.641111 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.641474 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.641916 13885 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:37.642817 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.642941 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.643664 13880 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:37.649629 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.658866 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.659561 13813 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:37.660358 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.661829 13805 raft_consensus.cc:1273] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:59:37.664212 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.666381 14479 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:37.642084 13887 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:37.667738 14479 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:37.664793 13804 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:37.687348 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.687441 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.689045 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.772603 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.773345 13880 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:37.773497 13885 raft_consensus.cc:1238] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20250623 01:59:37.774257 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.775385 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.775962 13880 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:37.776734 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.770850 13805 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:37.743196 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:37.782641 13879 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:37.728876 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:37.747686 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 82575.4us
I20250623 01:59:37.802631 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.628751 16574 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:37.816885 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.810279 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:37.834805 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 89886us
I20250623 01:59:37.742851 16438 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:37.847862 16574 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:37.848675 13804 raft_consensus.cc:2374] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6. Re-sending same reply.
I20250623 01:59:37.833899 16794 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:37.852788 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:37.841377 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.852986 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:37.858482 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:37.824563 16799 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250623 01:59:37.859123 16789 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.053s sys 0.103s
W20250623 01:59:37.864925 16789 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s user 0.053s sys 0.103s
I20250623 01:59:37.867535 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.869402 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.719522 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.865749 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.870589 13811 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:37.868356 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.878396 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.680529 16780 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:37.881116 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:37.894524 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 98194.5us
W20250623 01:59:37.895208 13854 leader_election.cc:336] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.803s (SENT)
I20250623 01:59:37.895006 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.898204 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.901098 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.902159 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.919085 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:37.919898 13885 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:37.926434 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.928442 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.931352 16794 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:37.931849 16794 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:37.937587 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:37.937698 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 48927us
I20250623 01:59:37.941840 13812 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:37.942521 13885 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:37.944968 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.910254 13854 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:37.947697 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.957589 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.957002 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.959189 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.959776 13810 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:37.924916 13953 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "afad1ff768904b11936d338d33df7998"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:37.961808 13953 raft_consensus.cc:604] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:37.962159 13953 raft_consensus.cc:612] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:37.962654 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:37.963956 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 68777us
I20250623 01:59:37.965621 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.969453 16666 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:37.970268 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.970835 13807 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:37.971778 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:37.972342 13955 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:37.974105 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.975622 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.977360 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.977653 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.978467 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.978696 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.979521 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.979643 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.980162 13880 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:37.980542 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.980945 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.981644 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.982208 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:37.954777 16806 log.cc:927] Time spent T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.088s user 0.004s sys 0.000s
I20250623 01:59:37.983817 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.992447 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.993515 13807 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:37.992332 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:37.995154 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.991204 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.997825 13882 raft_consensus.cc:2366] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:37.994800 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:37.999081 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:37.999845 13810 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:38.002360 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.005860 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.006768 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.006827 13885 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:38.008543 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.009599 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.010776 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.015991 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.022163 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.024331 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.024820 13883 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:38.027639 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.027824 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.028779 13960 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:38.028007 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.026232 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:38.013546 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250623 01:59:38.030819 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b39b05807ed544d388835cc364cb7274" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.032145 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.032939 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.033543 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.034442 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.034915 13885 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:38.035878 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "433978dce7d94bcb8968bc09583357c4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.037627 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.033618 13882 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:38.038301 13812 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:38.038576 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.039846 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.040906 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.046382 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.047238 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.047194 13804 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:38.052117 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.051177 13811 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:38.057451 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.058302 13805 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:38.060312 13812 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:38.063473 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.063841 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.064421 13880 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:38.063364 13925 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.072854 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.067914 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:38.089901 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 262363us
I20250623 01:59:38.105959 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:38.034277 16794 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.093706 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:38.119406 14508 raft_consensus.cc:1238] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:38.123502 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.126503 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.103655 16580 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:38.137665 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.137784 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.133817 14478 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.142418 14509 raft_consensus.cc:1238] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:38.137171 16580 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:38.146240 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.147099 14508 raft_consensus.cc:1238] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250623 01:59:38.063897 13883 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:38.144249 16580 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:59:38.150964 16812 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.166s user 0.002s sys 0.000s
W20250623 01:59:38.157836 13695 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2928af9e5df4f9eacf59f499ab96dc1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 85a58a834b6b4b0dbe197b3dad87e382 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:38.158283 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.159602 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.159700 13852 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.160212 13885 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:38.160912 13852 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.161125 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.163734 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.168098 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.172067 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.173833 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.177369 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.172866 13960 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:38.179263 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.179618 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.180260 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.149614 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6eba0cd4da75482ab2fcaad6bb1c17a9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:38.178335 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:38.184176 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.182751 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.184366 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.183398 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.186784 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819094ffb4314769ab2a06512232fc61" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.190496 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.175161 16574 raft_consensus.cc:2747] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:38.178336 16666 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:38.196933 16794 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:38.198196 13960 raft_consensus.cc:1238] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 2. Current term is 3. Ops: []
I20250623 01:59:38.198738 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:38.199925 13853 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.203496 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:38.207371 16811 log.cc:927] Time spent T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.305s user 0.002s sys 0.000s
I20250623 01:59:38.208528 13961 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250623 01:59:38.208882 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.209669 13957 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:38.204958 16580 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.210734 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:38.212044 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:38.212190 16580 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:38.212901 16580 consensus_queue.cc:1046] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250623 01:59:38.196969 16666 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:38.213904 16666 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:38.204999 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.195360 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:38.195744 16762 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:38.219846 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.228106 13959 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:38.228477 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.230700 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.232343 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.238209 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.239161 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.240227 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.243331 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:38.244156 13960 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:38.247180 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:38.234843 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.260509 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:38.276468 13779 connection.cc:232] RPC call timeout handler was delayed by -2.6085183559999905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19890000000000002s for a response
I20250623 01:59:38.276780 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:38.278121 13779 connection.cc:232] RPC call timeout handler was delayed by -2.5977183560000867s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
W20250623 01:59:38.279256 13779 connection.cc:232] RPC call timeout handler was delayed by -1.3847922549999794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20250623 01:59:38.279670 13779 connection.cc:232] RPC call timeout handler was delayed by -1.3514922549999255s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
W20250623 01:59:38.280011 13779 connection.cc:232] RPC call timeout handler was delayed by -1.3343922550000116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16080000000000003s for a response
W20250623 01:59:38.280496 13779 connection.cc:232] RPC call timeout handler was delayed by -1.258792255000003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
W20250623 01:59:38.282106 13779 connection.cc:232] RPC call timeout handler was delayed by -1.2317922549999594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
W20250623 01:59:38.313033 13779 connection.cc:232] RPC call timeout handler was delayed by -1.2038922549999143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1753s for a response
W20250623 01:59:38.313657 13779 connection.cc:232] RPC call timeout handler was delayed by -1.1453922549999334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
W20250623 01:59:38.314113 13779 connection.cc:232] RPC call timeout handler was delayed by -1.1138922549999961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18530000000000002s for a response
W20250623 01:59:38.328598 13779 connection.cc:232] RPC call timeout handler was delayed by -1.0769922549999364s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
I20250623 01:59:38.347573 14515 raft_consensus.cc:1238] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:38.369517 13779 leader_election.cc:400] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:38.374178 13779 leader_election.cc:403] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20250623 01:59:38.382822 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 136331us
W20250623 01:59:38.383808 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 250941us
W20250623 01:59:38.389094 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 231233us
W20250623 01:59:38.391422 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 241207us
I20250623 01:59:38.400668 13779 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:38.390878 16575 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
W20250623 01:59:38.441120 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 240099us
W20250623 01:59:38.440771 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 279970us
I20250623 01:59:38.443594 16580 raft_consensus.cc:3053] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:38.444022 16580 raft_consensus.cc:738] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:38.444837 16814 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20250623 01:59:38.444931 16580 consensus_queue.cc:260] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.446302 16580 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:38.446460 16815 log.cc:927] Time spent T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20250623 01:59:38.447528 16666 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:38.460763 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 272110us
I20250623 01:59:38.439913 16462 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:38.474844 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.487339 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:59:38.492076 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 92225us
W20250623 01:59:38.496934 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 96149.2us
W20250623 01:59:38.499603 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 103055us
I20250623 01:59:38.500561 14507 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 10. Current term is 11. Ops: []
I20250623 01:59:38.500283 16839 consensus_queue.cc:1046] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250623 01:59:38.484015 16462 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:38.506006 16462 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:38.506604 16836 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:38.507069 16836 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:38.507462 16836 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:38.501116 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:38.505236 16826 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:38.510730 16826 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:38.511147 16826 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:38.513947 16839 raft_consensus.cc:3053] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:38.514355 16839 raft_consensus.cc:738] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:38.514490 13926 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:38.515022 16762 consensus_queue.cc:1046] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:38.515398 16839 consensus_queue.cc:260] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.516129 16762 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:38.516513 16762 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:38.516819 16762 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:38.517518 16833 raft_consensus.cc:3053] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:38.517632 16462 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.518010 14511 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20250623 01:59:38.520794 16462 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:38.517984 16833 raft_consensus.cc:738] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:38.523332 14478 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.531138 13926 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:59:38.539362 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 147282us
W20250623 01:59:38.540127 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 154294us
W20250623 01:59:38.541850 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 46017.9us
W20250623 01:59:38.542544 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 54382.7us
I20250623 01:59:38.544420 16830 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250623 01:59:38.544641 13852 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.545842 13852 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.546015 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:38.514421 16666 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:38.516825 16839 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:38.558358 16609 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:38.558881 16609 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:38.558625 16828 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:38.559386 16609 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
W20250623 01:59:38.564436 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 45206.5us
I20250623 01:59:38.548817 16462 consensus_queue.cc:1046] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20250623 01:59:38.524724 16580 consensus_queue.cc:1046] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20250623 01:59:38.572816 13776 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.573549 13779 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:38.575305 16847 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:38.589844 16847 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:38.590329 16847 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:38.595561 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:38.605715 13954 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:38.605515 16840 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250623 01:59:38.609706 16833 consensus_queue.cc:260] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.611093 16833 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:38.613111 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 91276.1us
W20250623 01:59:38.618570 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 45930.4us
I20250623 01:59:38.620133 13954 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 11.
I20250623 01:59:38.625255 13776 leader_election.cc:400] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
W20250623 01:59:38.625344 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 79149.2us
I20250623 01:59:38.573463 16666 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20250623 01:59:38.621451 16575 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:38.618566 16462 raft_consensus.cc:2747] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:38.627627 16838 raft_consensus.cc:3053] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:38.632513 16838 raft_consensus.cc:738] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:38.636523 16645 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:38.637012 16645 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:38.637423 16645 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:38.643425 16847 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.635769 16836 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.645361 16848 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:38.645388 16838 consensus_queue.cc:260] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.647907 16838 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:38.648195 13776 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.648325 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52444 (request call id 275) took 9274 ms. Trace:
I20250623 01:59:38.648412 16826 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.649868 16836 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:38.650352 13709 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 6 to 8, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 8 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:38.667243 13887 raft_consensus.cc:1238] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:38.671420 16563 consensus_queue.cc:1046] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:38.672478 16563 raft_consensus.cc:3053] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:38.672824 16563 raft_consensus.cc:738] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:38.673604 16563 consensus_queue.cc:260] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.674960 16563 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:38.675469 16848 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:38.681623 16645 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.698252 16645 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:38.683424 16842 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:38.684232 13852 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.699518 16841 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:38.700183 16848 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:38.701220 16841 raft_consensus.cc:695] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:38.703372 16836 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:38.702883 16841 consensus_queue.cc:237] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.706182 13852 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:38.692165 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 44020.4us
I20250623 01:59:38.648869 13708 rpcz_store.cc:276] 0623 01:59:29.373577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:29.373731 (+ 154us) service_pool.cc:225] Handling call
0623 01:59:38.648238 (+9274507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:37.086625 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:37.129113 (+ 42488us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:37.129178 (+ 65us) write_op.cc:435] Acquired schema lock
0623 01:59:37.129198 (+ 20us) tablet.cc:660] Decoding operations
0623 01:59:37.152134 (+ 22936us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:37.152265 (+ 131us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:37.152309 (+ 44us) tablet.cc:683] Acquiring locks for 6 operations
0623 01:59:37.153269 (+ 960us) tablet.cc:699] Row locks acquired
0623 01:59:37.153303 (+ 34us) write_op.cc:260] PREPARE: finished
0623 01:59:37.153586 (+ 283us) write_op.cc:270] Start()
0623 01:59:37.153772 (+ 186us) write_op.cc:276] Timestamp: P: 1750643977153493 usec, L: 0
0623 01:59:37.153796 (+ 24us) op_driver.cc:348] REPLICATION: starting
0623 01:59:37.171263 (+ 17467us) log.cc:844] Serialized 2446 byte log entry
0623 01:59:38.520847 (+1349584us) op_driver.cc:464] REPLICATION: finished
0623 01:59:38.557270 (+ 36423us) write_op.cc:301] APPLY: starting
0623 01:59:38.557398 (+ 128us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:38.557591 (+ 193us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:38.557728 (+ 137us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:38.558681 (+ 953us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:38.559087 (+ 406us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0623 01:59:38.559155 (+ 68us) write_op.cc:312] APPLY: finished
0623 01:59:38.559482 (+ 327us) log.cc:844] Serialized 53 byte log entry
0623 01:59:38.559686 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:38.559803 (+ 117us) write_op.cc:454] Released schema lock
0623 01:59:38.559887 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:38.622656 (+ 62769us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:38.628548 (+ 5892us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5129212,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":36117,"apply.run_cpu_time_us":3237,"apply.run_wall_time_us":38070,"num_ops":6,"prepare.queue_time_us":167681,"prepare.run_cpu_time_us":161221,"prepare.run_wall_time_us":786252,"raft.queue_time_us":344592,"raft.run_cpu_time_us":5231,"raft.run_wall_time_us":62255,"replication_time_us":1366833,"thread_start_us":1139129,"threads_started":4,"wal-append.queue_time_us":813150}]]}
I20250623 01:59:38.671862 16826 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:38.709590 13776 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.710940 16849 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:38.711176 16842 consensus_queue.cc:1046] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:38.711751 16849 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:38.712255 16849 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:38.712565 16842 raft_consensus.cc:3053] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:38.713025 16842 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:38.713902 16842 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.715282 16842 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:38.710945 13776 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:38.717603 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 161524us
I20250623 01:59:38.718410 16847 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:38.723096 16838 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:38.723632 16838 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:38.723369 16762 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.724185 16838 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
W20250623 01:59:38.724781 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 63189.9us
I20250623 01:59:38.729521 13852 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.730329 13852 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:38.730765 16826 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:38.731033 16836 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:38.730337 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:38.731428 16836 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:38.732182 16836 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:38.732301 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.733218 13852 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.733596 16851 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:38.734011 16851 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:38.734376 16851 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
W20250623 01:59:38.735495 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 187316us
I20250623 01:59:38.731530 16826 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:38.740219 16840 raft_consensus.cc:3053] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:38.740655 16840 raft_consensus.cc:738] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:38.741719 16840 consensus_queue.cc:260] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.745143 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:38.745931 16575 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:38.746017 14511 raft_consensus.cc:2391] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 9: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:38.746764 13779 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:38.746852 16575 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:59:38.748736 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 209317us
W20250623 01:59:38.750520 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 209786us
W20250623 01:59:38.751205 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 200838us
I20250623 01:59:38.747571 16840 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:38.743407 16836 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.752429 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:38.761792 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.762158 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.757586 16851 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.764425 16852 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:38.764910 16852 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:38.765245 16852 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:38.766139 16851 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:38.767199 16845 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:38.751355 16575 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:38.778898 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.780000 13776 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:38.781359 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.781966 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.782207 13813 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:38.720456 16849 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.779434 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:38.784284 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.777009 16861 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:38.782738 13812 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:38.787825 13779 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:38.785912 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.790663 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.790804 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.792551 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:38.792774 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.758253 16833 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20250623 01:59:38.795800 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.785986 13925 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.795935 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.794440 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.795845 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.797885 13805 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:38.808341 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.811806 16762 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:38.812464 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:38.813540 13805 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:38.814679 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:38.788741 16861 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:38.823143 13776 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.796006 16854 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
W20250623 01:59:38.828713 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 68403.7us
I20250623 01:59:38.791499 16851 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20250623 01:59:38.813640 16832 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:38.830364 13926 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.817917 16822 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:38.832028 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:38.836155 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 48366.7us
I20250623 01:59:38.836861 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:38.837010 16864 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:38.837478 13808 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:38.837674 16864 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:38.838263 16864 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:38.796054 16848 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20250623 01:59:38.792551 16609 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.839515 13925 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.823738 16861 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:38.831336 16860 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:38.830350 16462 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:38.851668 13808 raft_consensus.cc:2466] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:38.815963 16828 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20250623 01:59:38.850440 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:38.821545 16841 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20250623 01:59:38.857499 16849 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:38.855633 13925 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:38.854414 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 51668us
I20250623 01:59:38.834055 16822 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:38.862372 13854 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:38.884382 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 46990.7us
I20250623 01:59:38.884650 16865 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:38.886006 16837 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250623 01:59:38.890095 13776 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.890558 16849 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:38.891032 16849 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:38.891434 16849 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:38.891511 13776 leader_election.cc:304] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.861588 16462 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:38.893939 16462 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:38.894474 16462 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:38.895025 16462 raft_consensus.cc:2747] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:38.895172 16852 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.893188 13776 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.901698 13776 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.901827 16645 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:38.875202 13925 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.906004 14515 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:38.854427 16860 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:38.855404 16854 raft_consensus.cc:695] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:38.915944 13805 raft_consensus.cc:2466] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:38.897183 16852 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:38.949828 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 74728.6us
I20250623 01:59:38.950573 13925 leader_election.cc:304] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.901737 16842 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:38.922371 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.952728 13880 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:38.902320 16836 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:38.874150 16822 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:38.959128 16870 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250623 01:59:38.962369 16862 log.cc:927] Time spent T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250623 01:59:38.962647 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:38.966390 16853 raft_consensus.cc:2747] T 6eba0cd4da75482ab2fcaad6bb1c17a9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:38.862504 16840 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250623 01:59:38.946197 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 44557.3us
I20250623 01:59:38.968983 13776 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.911356 16645 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:38.947438 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:38.929394 13854 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.950773 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:38.971909 13808 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:38.949906 13853 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.921353 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:38.859138 16838 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.953698 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:38.944854 16869 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:38.953194 13925 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.946593 16860 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:38.950335 16833 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:38.975917 13805 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:38.976531 13925 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:38.979748 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 119) took 6557 ms. Trace:
I20250623 01:59:38.983485 13811 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:38.983875 16838 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:38.984812 16851 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:38.986016 16609 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:38.969151 16840 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:38.987017 16840 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:38.991067 16849 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.992311 16575 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:38.973480 16645 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:38.993626 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 64363.7us
I20250623 01:59:39.022359 13854 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.006911 13779 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:39.010951 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 56752.4us
W20250623 01:59:38.997332 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 155057us
I20250623 01:59:39.026883 13926 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:39.027889 13852 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.993459 13810 raft_consensus.cc:1238] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250623 01:59:39.002524 13776 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.013149 13808 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:39.014948 13925 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:39.046901 13805 raft_consensus.cc:2466] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
W20250623 01:59:39.062526 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 55693.3us
W20250623 01:59:39.064301 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 61899.6us
I20250623 01:59:39.065714 13776 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.063540 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:59:39.068480 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 46216.2us
I20250623 01:59:39.069419 13854 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:39.079151 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 64271.7us
I20250623 01:59:38.980304 13709 rpcz_store.cc:276] 0623 01:59:32.422513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:32.422664 (+ 151us) service_pool.cc:225] Handling call
0623 01:59:38.979666 (+6557002us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:38.740264 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:38.740796 (+ 532us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:38.740823 (+ 27us) write_op.cc:435] Acquired schema lock
0623 01:59:38.740834 (+ 11us) tablet.cc:660] Decoding operations
0623 01:59:38.741176 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:38.741251 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:38.741287 (+ 36us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:59:38.741490 (+ 203us) tablet.cc:699] Row locks acquired
0623 01:59:38.741504 (+ 14us) write_op.cc:260] PREPARE: finished
0623 01:59:38.741650 (+ 146us) write_op.cc:270] Start()
0623 01:59:38.741749 (+ 99us) write_op.cc:276] Timestamp: P: 1750643978741598 usec, L: 0
0623 01:59:38.741765 (+ 16us) op_driver.cc:348] REPLICATION: starting
0623 01:59:38.742389 (+ 624us) log.cc:844] Serialized 591 byte log entry
0623 01:59:38.790764 (+ 48375us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b18003475c0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e74c16ad 00007f41e74c39aa 00007f41e74c3938 00007f41e74c38b8 00007f41e74c3873 00007f41e74c2a8c 00007f41e9865ff6 00007f41e9860848 00007f41e3aaaef0
0623 01:59:38.795888 (+ 5124us) op_driver.cc:464] REPLICATION: finished
0623 01:59:38.821604 (+ 25716us) write_op.cc:301] APPLY: starting
0623 01:59:38.825658 (+ 4054us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:38.831550 (+ 5892us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:38.833830 (+ 2280us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:38.855718 (+ 21888us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:38.877584 (+ 21866us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:59:38.890664 (+ 13080us) write_op.cc:312] APPLY: finished
0623 01:59:38.890866 (+ 202us) log.cc:844] Serialized 23 byte log entry
0623 01:59:38.891023 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:38.891114 (+ 91us) write_op.cc:454] Released schema lock
0623 01:59:38.891163 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:38.891592 (+ 429us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:38.891621 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":950461,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":7659,"apply.run_cpu_time_us":2038,"apply.run_wall_time_us":69939,"num_ops":1,"prepare.queue_time_us":67715,"prepare.run_cpu_time_us":2535,"prepare.run_wall_time_us":14780,"replication_time_us":14217,"spinlock_wait_cycles":9423616,"thread_start_us":72025,"threads_started":1}]]}
I20250623 01:59:39.082130 13854 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.067764 13852 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:39.006343 16859 log.cc:927] Time spent T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20250623 01:59:38.997016 13710 catalog_manager.cc:5582] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 5, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:39.006376 16575 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.006428 16853 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:38.995869 16851 raft_consensus.cc:695] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:39.015156 16833 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:39.103876 16874 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:39.104719 16852 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250623 01:59:39.106488 16872 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.207s user 0.000s sys 0.004s
W20250623 01:59:39.107625 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:39.108856 13852 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:38.993968 16867 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:39.002729 16869 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:39.113879 16867 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:39.114365 16867 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:39.100438 16848 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:38.985191 16861 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.121691 16851 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:39.124511 16861 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.094172 13925 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:39.072531 16873 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:39.072535 16838 raft_consensus.cc:2747] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:59:39.092064 13776 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.085702 13779 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
W20250623 01:59:39.136070 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 50487us
I20250623 01:59:39.136863 13779 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.078748 16609 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:39.137642 16609 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:39.138051 16609 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:39.072103 16666 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:39.095850 13710 catalog_manager.cc:5582] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:38.993247 16849 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.013329 16860 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:39.142660 13804 raft_consensus.cc:1238] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:39.144006 16860 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:39.131835 13854 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:39.089536 16823 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250623 01:59:39.134342 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 107571us
I20250623 01:59:39.144763 16823 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:39.087194 16828 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:39.145663 16828 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:39.145226 16823 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:39.146173 16666 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:39.146703 16666 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:39.147090 16666 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:39.145156 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:39.146144 16828 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:39.146215 13854 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:39.150444 16848 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:39.092996 16837 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:39.151008 13854 leader_election.cc:304] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
W20250623 01:59:39.153494 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 44715.6us
W20250623 01:59:39.152792 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 58702.1us
I20250623 01:59:39.002943 16822 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.155956 13926 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.143474 13710 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 3 to 4, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2), now has a pending config: VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1), VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2), VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3), VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 4 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: NON_VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: true } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:39.127995 16853 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:39.170183 13779 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.171533 13852 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.152495 16852 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:39.153100 16575 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:39.175637 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 83641.4us
I20250623 01:59:39.183279 13776 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.158316 16839 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:39.152333 16848 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:39.005543 16854 consensus_queue.cc:237] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.193876 16853 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.163909 16861 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:39.202347 13926 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.203573 13852 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.212605 16848 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:39.222817 16822 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:39.223520 16822 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:39.225576 16836 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:39.217298 13776 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.218888 13926 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:39.219671 13852 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:39.204090 16864 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.205675 16873 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:39.172501 16842 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250623 01:59:39.232679 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 81737.5us
I20250623 01:59:39.232833 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:39.172336 16856 raft_consensus.cc:2747] T b39b05807ed544d388835cc364cb7274 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:39.157866 16762 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:39.205025 13779 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.174782 16837 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:39.242969 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.204139 16847 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:39.233434 13854 leader_election.cc:304] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.237088 13852 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:39.175204 16852 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:39.249715 13852 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:39.260247 13779 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:39.261902 13779 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
W20250623 01:59:39.262882 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 44048.9us
I20250623 01:59:39.263602 13926 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:39.217643 16861 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:39.212853 16839 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:39.218648 16867 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.246748 16762 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:39.238286 16849 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:39.273299 13779 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.246409 16871 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:39.246001 16860 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:39.246153 16870 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:39.282611 13852 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:39.280709 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 144892us
W20250623 01:59:39.279263 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 62047.6us
I20250623 01:59:39.262501 16852 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:59:39.311185 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 134175us
I20250623 01:59:39.217089 16645 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.265229 16836 raft_consensus.cc:2802] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:39.264509 16855 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:39.275779 16837 raft_consensus.cc:2747] T 2540092c63cb4d0cbe07ae4018f99f46 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:59:39.312839 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 49309.1us
I20250623 01:59:39.267307 16868 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:39.321440 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.321370 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:39.322414 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.278831 16842 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:39.323572 13887 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 7: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:39.324235 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.285794 16847 raft_consensus.cc:2802] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:39.278097 16762 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:39.275566 16839 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
W20250623 01:59:39.312204 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 78883.1us
I20250623 01:59:39.329174 13854 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.278030 16863 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:39.334494 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.321377 13926 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:59:39.315960 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:39.284332 16865 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:39.325897 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.281829 16856 raft_consensus.cc:2747] T 433978dce7d94bcb8968bc09583357c4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:39.318027 16841 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:39.334276 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.330322 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.277920 16874 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20250623 01:59:39.312601 16870 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:39.342281 13882 raft_consensus.cc:1238] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:39.340672 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.274070 16834 consensus_queue.cc:1046] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20250623 01:59:39.340193 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:39.318871 16855 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:39.270488 16869 consensus_queue.cc:1046] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20250623 01:59:39.348734 13926 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:39.326531 16836 raft_consensus.cc:695] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:39.348570 16834 raft_consensus.cc:3053] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:39.346427 16845 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:39.368822 13926 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:39.346177 13852 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:39.356007 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.357708 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.357868 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.349828 16867 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.369998 13776 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.358570 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.359138 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.359799 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:39.359284 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.334939 16863 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:39.388285 16826 raft_consensus.cc:2802] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:39.388410 16863 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:39.388599 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.388818 16826 raft_consensus.cc:695] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:39.313699 16840 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.389838 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.365154 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.390877 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.391402 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.391810 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.320130 16832 consensus_queue.cc:1046] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20250623 01:59:39.367596 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.361160 16855 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:39.370862 13955 raft_consensus.cc:2408] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250623 01:59:39.372368 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.373966 16871 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:39.367107 16575 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:39.375483 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.376433 16833 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:39.343995 16847 raft_consensus.cc:695] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:39.358467 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.320459 16609 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.377274 16869 raft_consensus.cc:3053] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:59:39.379768 16864 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.366434 16834 raft_consensus.cc:738] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:39.414722 16869 raft_consensus.cc:738] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:39.418833 16834 consensus_queue.cc:260] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:39.419499 16869 consensus_queue.cc:260] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.420302 16834 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:39.420841 16869 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:39.400104 16873 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:39.409242 16833 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:39.422118 16833 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:39.405607 16849 consensus_queue.cc:1046] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20250623 01:59:39.427408 16849 raft_consensus.cc:3053] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:39.427804 16849 raft_consensus.cc:738] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:39.344662 16870 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:39.429411 16867 raft_consensus.cc:3053] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:39.429838 16867 raft_consensus.cc:738] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:39.406250 16871 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:39.433530 16884 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:39.435683 16609 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.436735 16847 consensus_queue.cc:237] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.437713 16836 consensus_queue.cc:237] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.381951 13926 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:59:39.441682 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 59797us
I20250623 01:59:39.442605 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:39.393018 13957 raft_consensus.cc:1238] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:39.370692 13961 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:39.359138 16841 raft_consensus.cc:695] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:39.400890 13776 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.348523 16889 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:39.374572 16828 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.404832 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.447685 13954 raft_consensus.cc:1238] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250623 01:59:39.415689 13956 raft_consensus.cc:2391] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:39.393483 16666 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
W20250623 01:59:39.470675 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 68920.7us
I20250623 01:59:39.471601 13776 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.473973 13926 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.475596 13812 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 9. Current term is 10. Ops: []
I20250623 01:59:39.478457 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.479072 13812 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:39.478974 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.425114 16852 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.456306 16889 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:39.389850 16826 consensus_queue.cc:237] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.428757 16849 consensus_queue.cc:260] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.510075 13885 raft_consensus.cc:1238] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:39.470566 16645 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:39.500257 13926 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.513413 13809 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20250623 01:59:39.519464 13809 pending_rounds.cc:85] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 9
I20250623 01:59:39.473229 16575 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:39.519726 13809 pending_rounds.cc:107] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20250623 01:59:39.555423 13809 raft_consensus.cc:2887] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:39.520954 16888 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250623 01:59:39.521855 16885 log.cc:927] Time spent T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20250623 01:59:39.430650 16867 consensus_queue.cc:260] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.512121 16889 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:39.524585 16855 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.525727 16849 raft_consensus.cc:3058] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:39.572600 16852 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.531916 16840 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:39.532423 16841 consensus_queue.cc:237] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.456995 16848 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.534495 16870 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.542754 13926 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:39.543409 16884 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250623 01:59:39.590297 16812 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20250623 01:59:39.545387 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.592321 13811 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:39.546564 13810 raft_consensus.cc:1238] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:39.547550 16762 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.553165 16833 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.562170 16860 consensus_queue.cc:1046] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:39.563745 16462 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:39.606547 16462 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.608947 16462 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.564689 13804 raft_consensus.cc:1238] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:39.440363 16823 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.617537 16823 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:39.497597 16873 raft_consensus.cc:2747] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:39.577716 16867 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:39.620245 16871 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.517151 16839 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.630892 16839 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:39.632150 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.578740 16848 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.630897 16871 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.567353 16863 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.653825 13813 raft_consensus.cc:2391] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:39.656121 13807 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:39.665809 16574 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:39.666683 16574 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:39.668130 16686 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:39.668635 16686 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:39.670559 16686 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:39.596307 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 53628.5us
I20250623 01:59:39.604017 16762 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:39.605005 16860 raft_consensus.cc:3053] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:39.672389 16860 raft_consensus.cc:738] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:39.672638 13926 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:39.673774 13926 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.673460 16860 consensus_queue.cc:260] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.674535 16823 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250623 01:59:39.675096 16860 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:39.675348 13926 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:39.606015 16891 log.cc:927] Time spent T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250623 01:59:39.611354 13961 raft_consensus.cc:1238] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
I20250623 01:59:39.610932 16833 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:39.613695 14515 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250623 01:59:39.614420 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:39.612818 16828 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.578181 13884 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: []
W20250623 01:59:39.622277 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:39.680617 16828 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:39.686035 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:39.686830 13813 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:39.623903 16870 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.690434 16873 consensus_queue.cc:1046] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:39.666044 13880 raft_consensus.cc:1238] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 10. Current term is 11. Ops: []
I20250623 01:59:39.674125 16839 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:39.678968 13886 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:39.564023 16666 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 election: Requested vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781)
I20250623 01:59:39.693950 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.705579 13956 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:39.580646 13812 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:39.711113 13811 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:39.711043 14479 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:59:39.719161 13776 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:39.719422 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 44163.9us
I20250623 01:59:39.720660 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:39.695751 13779 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.573s (SENT) [suppressed 121 similar messages]
I20250623 01:59:39.741256 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:39.741969 13811 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
I20250623 01:59:39.718731 16823 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:39.757004 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:39.749946 14479 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:59:39.725062 16871 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:39.735185 16834 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:39.733740 16822 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:39.700093 16864 consensus_queue.cc:1046] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20250623 01:59:39.789232 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.792101 13954 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 8.
I20250623 01:59:39.781356 16871 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:39.785620 16834 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:39.816087 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.816958 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.834477 13954 raft_consensus.cc:2391] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:39.817221 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.818531 16882 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:39.840876 13954 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:39.842352 13954 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:39.836284 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.795960 16837 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:39.790280 16574 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:39.839205 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.843639 13957 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 9: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:39.844190 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:39.847816 13961 raft_consensus.cc:1238] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 10. Current term is 11. Ops: []
I20250623 01:59:39.850728 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.780423 16889 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:39.834506 16871 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:39.838402 16834 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:39.859740 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.860785 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.839884 16882 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:39.863428 13957 raft_consensus.cc:2391] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 5: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:39.872452 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.874697 13954 raft_consensus.cc:1238] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:39.880561 13886 raft_consensus.cc:1238] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20250623 01:59:39.882263 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.885421 13956 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:39.886595 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.887780 13954 raft_consensus.cc:1238] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:39.888442 13960 raft_consensus.cc:1238] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250623 01:59:39.889015 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:59:39.889603 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 193879us
W20250623 01:59:39.893057 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:39.895222 13812 raft_consensus.cc:1238] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:39.896560 13804 raft_consensus.cc:1238] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:39.897493 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.898003 13812 raft_consensus.cc:1238] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:39.898803 13810 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250623 01:59:39.899192 13809 raft_consensus.cc:1238] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:39.899899 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:39.900748 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 414056us
I20250623 01:59:39.901397 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.902364 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.904644 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.905037 13812 raft_consensus.cc:1238] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:39.906384 13810 raft_consensus.cc:2391] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 9: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:39.902980 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.909904 13883 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:39.910030 13957 raft_consensus.cc:2391] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 5: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:39.914378 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.914572 13962 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 10. Current term is 11. Ops: []
I20250623 01:59:39.916267 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.916682 13883 raft_consensus.cc:1238] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:39.916750 13879 raft_consensus.cc:1238] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
W20250623 01:59:39.920415 13959 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.027s sys 0.055s
W20250623 01:59:39.921162 13959 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.528s user 0.027s sys 0.056s
W20250623 01:59:39.921083 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 75465.8us
I20250623 01:59:39.913895 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.927124 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.918473 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.934494 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.941664 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:39.915596 13925 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:39.944409 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 43797.4us
I20250623 01:59:39.944774 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.945125 13854 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.945123 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.946343 13811 raft_consensus.cc:1238] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20250623 01:59:39.946822 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.951241 13880 raft_consensus.cc:2391] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:39.884990 16882 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:39.953477 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.942461 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.961856 13804 raft_consensus.cc:2391] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:39.959111 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.963649 13808 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:39.965215 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:39.966316 13962 raft_consensus.cc:2391] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:39.968302 13808 raft_consensus.cc:1238] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4. Ops: []
W20250623 01:59:39.974499 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 58964.9us
I20250623 01:59:39.977205 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.979167 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.980535 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.987846 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:39.989409 13779 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:39.997357 13807 raft_consensus.cc:1238] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
W20250623 01:59:39.999068 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 160973us
I20250623 01:59:39.999441 13962 raft_consensus.cc:1238] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:39.965476 16462 raft_consensus.cc:3053] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:40.026070 16869 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250623 01:59:40.117871 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.022308 16870 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.107s
I20250623 01:59:40.122599 13852 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:40.134166 16884 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250623 01:59:40.023587 16880 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
W20250623 01:59:40.140152 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:39.990489 16863 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:40.031472 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 86421.1us
I20250623 01:59:40.181008 13854 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:39.960980 16855 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:40.027552 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:39.976740 16823 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:40.027846 16867 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:40.183487 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:40.162122 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.031061 16847 consensus_queue.cc:1046] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.125s
I20250623 01:59:40.026597 16873 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:40.161664 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:40.031299 16848 consensus_queue.cc:1046] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.128s
W20250623 01:59:40.179350 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 56871.9us
I20250623 01:59:40.133585 16826 consensus_queue.cc:1046] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250623 01:59:40.134362 16854 consensus_queue.cc:1046] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20250623 01:59:40.164144 16869 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:40.197113 13852 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:40.194716 13962 raft_consensus.cc:1238] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:40.164788 16833 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:40.117875 16904 consensus_queue.cc:1046] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.151s
I20250623 01:59:40.210114 16904 raft_consensus.cc:3053] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:40.210474 16904 raft_consensus.cc:738] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:40.211375 16904 consensus_queue.cc:260] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.160254 16462 raft_consensus.cc:738] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:40.212627 16904 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.213389 16886 consensus_queue.cc:1046] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20250623 01:59:40.213513 16462 consensus_queue.cc:260] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.214637 16462 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:40.214985 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.190364 16823 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:40.216095 16823 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
W20250623 01:59:40.218772 16905 log.cc:927] Time spent T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20250623 01:59:40.220283 16901 log.cc:927] Time spent T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250623 01:59:40.221782 16869 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.226922 16913 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:40.227521 16913 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:40.203053 13854 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:40.207499 16894 log.cc:927] Time spent T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.308s user 0.000s sys 0.001s
I20250623 01:59:40.233438 16867 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:40.234570 16848 raft_consensus.cc:3053] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:40.235028 16848 raft_consensus.cc:738] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.235299 16822 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:40.235785 16822 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:40.235849 16848 consensus_queue.cc:260] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:40.237107 16849 consensus_meta.cc:287] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.667s user 0.018s sys 0.001s
I20250623 01:59:40.233764 16847 raft_consensus.cc:3053] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:40.186669 16856 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:40.244709 13854 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:40.250555 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:40.252326 13805 raft_consensus.cc:2466] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:40.260999 16898 raft_consensus.cc:3053] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Stepping down as leader of term 9
I20250623 01:59:40.261557 16898 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.262367 16898 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.263629 16898 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:40.265609 16919 consensus_queue.cc:1046] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:40.266774 16919 raft_consensus.cc:3053] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:40.267093 16919 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:40.267835 16822 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.241564 16907 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:40.269960 16907 raft_consensus.cc:695] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.270762 16907 consensus_queue.cc:237] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.271585 16871 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.238087 16912 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:40.274535 16912 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:40.241398 13852 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:40.277536 16828 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:40.278198 16828 raft_consensus.cc:695] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.279211 16828 consensus_queue.cc:237] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.279903 16870 consensus_queue.cc:1046] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:40.280871 16870 raft_consensus.cc:3053] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:40.281174 16870 raft_consensus.cc:738] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.240267 16886 raft_consensus.cc:3053] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:40.281908 16870 consensus_queue.cc:260] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.282474 16886 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.283051 16870 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.283273 16886 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.284380 16886 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:40.237277 16848 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:40.239410 16847 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.286896 16899 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:40.287051 16847 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.287662 16856 raft_consensus.cc:695] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.288270 16847 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:40.288522 16856 consensus_queue.cc:237] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.289460 13956 raft_consensus.cc:2466] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 11.
W20250623 01:59:40.290326 16893 log.cc:927] Time spent T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.528s user 0.002s sys 0.001s
I20250623 01:59:40.242198 16855 raft_consensus.cc:3053] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:40.293216 13959 raft_consensus.cc:1238] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20250623 01:59:40.239885 16851 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250623 01:59:40.318518 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 63985.3us
I20250623 01:59:40.320101 16867 consensus_queue.cc:1046] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:40.320487 16904 consensus_queue.cc:1046] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:40.320978 16871 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.321195 16867 raft_consensus.cc:3053] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Stepping down as leader of term 3
I20250623 01:59:40.321405 16904 raft_consensus.cc:3053] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:40.237877 16837 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:40.321667 16867 raft_consensus.cc:738] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.321978 16837 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:40.322116 16904 raft_consensus.cc:738] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:40.322638 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.322571 16867 consensus_queue.cc:260] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.297643 16855 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:40.321997 16822 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:40.323921 16867 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:40.323777 16904 consensus_queue.cc:260] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.267913 16919 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.324028 16855 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:40.331359 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 86722.9us
I20250623 01:59:40.333811 13854 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
W20250623 01:59:40.294596 13779 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:40.334939 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:40.335726 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.337494 16840 consensus_queue.cc:1046] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:40.338881 16645 raft_consensus.cc:3053] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:40.348408 13807 raft_consensus.cc:1273] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:40.349237 13804 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:40.351253 13813 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:40.325250 16919 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:40.346904 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.355845 13811 raft_consensus.cc:1273] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250623 01:59:40.339519 16903 log.cc:927] Time spent T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20250623 01:59:40.353237 16666 raft_consensus.cc:2747] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:40.363265 16866 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:40.363987 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:40.364136 16900 consensus_queue.cc:1035] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:40.365242 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:40.365877 13813 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 11.
I20250623 01:59:40.353423 16822 consensus_queue.cc:1046] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20250623 01:59:40.366663 16823 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.367559 16822 raft_consensus.cc:3053] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:40.367930 16822 raft_consensus.cc:738] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:40.369953 16834 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:40.372022 16834 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.374284 16869 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.346995 16851 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:40.375408 16851 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.376686 16869 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.353670 16904 raft_consensus.cc:3058] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:40.389885 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:40.390512 13805 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.381875 13961 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:40.383566 13807 raft_consensus.cc:1238] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:40.393677 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.395944 13804 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250623 01:59:40.396202 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:40.398286 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.398429 13852 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:40.400377 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 47672.7us
I20250623 01:59:40.401822 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:40.397153 16762 consensus_queue.cc:1046] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20250623 01:59:40.345037 16889 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:40.368736 16822 consensus_queue.cc:260] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.344467 16645 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:40.404388 16888 raft_consensus.cc:3053] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:40.409615 13806 raft_consensus.cc:1238] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:40.378824 16912 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.377301 16851 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.411428 13805 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:40.439349 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:40.440194 13811 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:40.434090 13807 raft_consensus.cc:1273] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:40.401432 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:40.445155 14479 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
W20250623 01:59:40.452517 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 51216.7us
I20250623 01:59:40.415215 16826 raft_consensus.cc:2802] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:40.458225 13852 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:40.417606 16888 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:40.459360 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 280083us
I20250623 01:59:40.446744 16889 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:40.467516 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 482421us
W20250623 01:59:40.468375 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 75397.4us
I20250623 01:59:40.454615 16822 raft_consensus.cc:3058] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:40.452108 16851 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:40.405134 16853 consensus_queue.cc:1046] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20250623 01:59:40.454443 16912 raft_consensus.cc:2802] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:40.431699 16836 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20250623 01:59:40.469302 13779 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:40.490127 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 466285us
I20250623 01:59:40.499116 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:40.474159 16899 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:40.500025 13883 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 9.
I20250623 01:59:40.477075 16913 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.474565 13925 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.475606 13884 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:40.472652 16826 raft_consensus.cc:695] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.472204 13854 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:40.508893 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:40.510156 13956 raft_consensus.cc:3058] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.510316 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.471455 16889 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.518172 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.519639 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:40.491600 16912 raft_consensus.cc:695] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
W20250623 01:59:40.520279 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 48169.4us
I20250623 01:59:40.518980 13953 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 9.
W20250623 01:59:40.521322 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 46797.6us
I20250623 01:59:40.520203 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 2410) took 1149 ms. Trace:
I20250623 01:59:40.482864 16580 consensus_queue.cc:1046] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20250623 01:59:40.483620 16841 consensus_queue.cc:1035] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20250623 01:59:40.523870 13955 rpcz_store.cc:276] 0623 01:59:39.363121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:39.406112 (+ 42991us) service_pool.cc:225] Handling call
0623 01:59:39.440516 (+ 34404us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:39.472709 (+ 32193us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:39.472744 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:39.492780 (+ 20036us) log.cc:844] Serialized 26 byte log entry
0623 01:59:39.920677 (+427897us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:59:39.991676 (+ 70999us) log.cc:844] Serialized 14 byte log entry
0623 01:59:40.163656 (+171980us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:40.163834 (+ 178us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:40.284312 (+120478us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b18001e3100. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b4061f 00007f41e0b428c1 00007f41e0b42879 00007f41e0b42809 00007f41e0b427d1 00007f41e0b41acd 00007f41e9b9b124 00007f41e9b9af59 00007f41e0b23646
0623 01:59:40.288680 (+ 4368us) spinlock_profiling.cc:243] Waited 8.01 ms on lock 0x7b18001e3100. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b404f2 00007f41e0b428c1 00007f41e0b42879 00007f41e0b42809 00007f41e0b427d1 00007f41e0b41acd 00007f41e9b9b124 00007f41e9b9af59 00007f41e0b23646
0623 01:59:40.509907 (+221227us) raft_consensus.cc:1683] finished
0623 01:59:40.509930 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:40.512752 (+ 2822us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8386,"prepare.queue_time_us":306896,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":150017,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":53334,"spinlock_wait_cycles":28682368,"thread_start_us":506103,"threads_started":3,"wal-append.queue_time_us":591525}
I20250623 01:59:40.520274 13882 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 10.
I20250623 01:59:40.526258 13925 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.496897 16823 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:40.503623 16899 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:40.527168 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.531520 13779 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.535195 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.536011 13953 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 11.
I20250623 01:59:40.476356 16645 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.509352 16922 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:40.545194 13956 raft_consensus.cc:2466] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
W20250623 01:59:40.563099 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 90665.1us
I20250623 01:59:40.550872 13779 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.522719 16666 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:40.559748 14481 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:40.575390 14481 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:40.530434 16899 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:40.565761 13882 raft_consensus.cc:1238] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 10. Current term is 11. Ops: []
W20250623 01:59:40.566419 16896 log.cc:927] Time spent T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.804s user 0.004s sys 0.000s
I20250623 01:59:40.526836 16927 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:40.578498 13779 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.583751 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.559782 16860 raft_consensus.cc:3053] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:40.557842 16823 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:40.562361 16645 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.546555 16873 consensus_queue.cc:1046] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20250623 01:59:40.622896 13808 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:40.557412 16869 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:40.586815 16575 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:40.581990 16849 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:40.562695 16837 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.596632 16462 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:40.599701 16866 consensus_queue.cc:1046] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:40.599975 16845 consensus_queue.cc:1046] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:40.589556 16563 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250623 01:59:40.600509 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:40.606122 16928 consensus_queue.cc:1046] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250623 01:59:40.588146 16926 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:40.609673 16912 consensus_queue.cc:237] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:40.610507 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:40.659144 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.611605 16833 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:40.548775 16929 consensus_queue.cc:1046] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20250623 01:59:40.590360 16860 raft_consensus.cc:738] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:40.653815 16563 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:59:40.662465 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 78822us
W20250623 01:59:40.663219 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 62799.8us
W20250623 01:59:40.674592 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 91193us
W20250623 01:59:40.663841 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 137639us
I20250623 01:59:40.675591 13925 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.584833 16826 consensus_queue.cc:237] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.675917 13779 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.655951 16926 raft_consensus.cc:491] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:40.680830 13854 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:40.678769 13925 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.666597 16866 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:40.660254 16666 consensus_queue.cc:1046] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20250623 01:59:40.674381 16563 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:40.684913 13925 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.672894 16856 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:40.692222 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.682718 16926 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:40.673485 16848 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.014s
I20250623 01:59:40.703147 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.684455 16927 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:40.705845 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:40.707206 16930 log.cc:927] Time spent T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250623 01:59:40.707290 16884 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:40.707827 16884 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:40.707990 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.709157 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.710014 13885 raft_consensus.cc:2408] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
W20250623 01:59:40.696220 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.713798 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.715263 16849 consensus_queue.cc:1046] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:40.716351 16873 raft_consensus.cc:3053] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:40.694547 16856 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.716728 16873 raft_consensus.cc:738] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:40.717406 13887 raft_consensus.cc:1238] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:40.718168 16856 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.717762 16873 consensus_queue.cc:260] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.718585 13879 raft_consensus.cc:1238] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:40.719280 16873 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:40.719434 13779 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.721249 16575 raft_consensus.cc:3053] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:40.721546 16943 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:40.721665 16575 raft_consensus.cc:738] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:40.722329 16943 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250623 01:59:40.722568 16575 consensus_queue.cc:260] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.723884 16575 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:40.724839 13779 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.689062 16853 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:40.728976 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.727620 13779 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.682857 16847 consensus_queue.cc:1046] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20250623 01:59:40.689181 16889 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:40.691973 16886 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:40.695704 16939 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:40.736558 13779 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:40.741750 16940 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250623 01:59:40.742506 16933 log.cc:927] Time spent T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20250623 01:59:40.742416 13956 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:40.743088 13956 raft_consensus.cc:604] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:40.743424 13956 raft_consensus.cc:612] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:40.743889 16855 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:59:40.749619 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.750444 16900 raft_consensus.cc:3053] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:40.750977 16900 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.751765 16900 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.702913 16860 consensus_queue.cc:260] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:40.734525 16863 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250623 01:59:40.733783 16882 consensus_meta.cc:287] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Time spent flushing consensus metadata: real 0.749s user 0.006s sys 0.004s
I20250623 01:59:40.724401 16851 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:40.757699 16837 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:40.753485 16939 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.783890 16939 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.784709 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:40.754343 16860 raft_consensus.cc:3058] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:40.786863 13708 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 8 to 11, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 11 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:40.790014 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.791013 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:40.791671 13956 raft_consensus.cc:2391] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 7: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:40.789124 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.795542 13955 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:40.796327 13955 raft_consensus.cc:604] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:40.796793 13955 raft_consensus.cc:612] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:40.792495 13708 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 4 to 5, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1), VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2), VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3), VOTER 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 4 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:40.798673 16822 consensus_queue.cc:1046] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20250623 01:59:40.805662 13879 raft_consensus.cc:1238] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:40.757040 16847 raft_consensus.cc:3053] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:40.758610 16943 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:40.754622 16762 raft_consensus.cc:3053] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:40.753285 16900 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.818465 16822 raft_consensus.cc:3053] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:40.818557 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:40.819252 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:40.821039 13806 raft_consensus.cc:1238] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:40.846487 13807 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 3.
I20250623 01:59:40.777833 16869 raft_consensus.cc:3053] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Stepping down as leader of term 7
W20250623 01:59:40.822271 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.848007 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:40.822690 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.823251 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.738272 16928 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:40.816431 16940 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:40.828202 16563 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:40.828907 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:40.829557 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:40.798936 16837 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20250623 01:59:40.796986 16864 consensus_queue.cc:1046] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20250623 01:59:40.838444 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.840536 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.858544 13884 raft_consensus.cc:2408] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250623 01:59:40.841158 13961 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:40.753399 16938 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:40.842999 13880 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:40.839169 16907 raft_consensus.cc:3053] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:40.867019 16907 raft_consensus.cc:738] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.867918 16907 consensus_queue.cc:260] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.868886 16907 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.807825 16922 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:40.849527 13955 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 3.
I20250623 01:59:40.851805 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.852030 13779 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.871796 13779 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.872686 13779 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:40.845818 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:59:40.852669 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.853948 13925 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.821851 16847 raft_consensus.cc:738] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:40.835618 13883 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:40.843876 16762 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:40.844204 16871 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:40.860541 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:40.860911 13811 raft_consensus.cc:2391] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 5: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:40.796784 16899 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.846131 16822 raft_consensus.cc:738] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:40.885599 16823 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:40.886107 16563 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 12 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.886395 16823 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:40.863126 13882 raft_consensus.cc:1238] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:40.886902 16823 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:40.885133 16822 consensus_queue.cc:260] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.887629 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:40.888213 13953 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:40.888381 16822 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:40.863422 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:40.889609 13813 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:40.855763 16943 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:40.863909 16864 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:40.891187 16864 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:40.891614 16847 consensus_queue.cc:260] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.891975 16864 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.865451 16940 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:40.874459 13925 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.895215 13810 raft_consensus.cc:1238] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:40.896222 13925 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.874444 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:40.875491 16928 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:40.900918 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:40.901556 13887 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:40.826651 16882 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:40.883723 16871 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.848143 16869 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:40.896962 13809 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:40.897615 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:40.898854 16904 consensus_queue.cc:1046] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:40.880846 16462 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:40.877192 16929 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:40.881709 13926 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:40.908239 13953 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 12.
I20250623 01:59:40.911296 13925 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.919241 13888 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:40.909248 13708 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 7 to 8, term changed from 8 to 11, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to <none>, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New cstate: current_term: 11 committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:40.918354 13882 raft_consensus.cc:3058] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 12
I20250623 01:59:40.920549 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:40.921941 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 241219us
I20250623 01:59:40.922530 13854 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:40.912746 16839 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250623 01:59:40.923053 16840 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:59:40.939440 16840 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:40.939817 16840 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:40.943892 13854 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:40.945133 13882 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 12.
I20250623 01:59:40.945673 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:40.945799 16899 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.923573 16869 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.943755 16836 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:40.924391 14515 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 9: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:40.924243 16904 consensus_queue.cc:1046] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:40.920315 16462 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:40.925232 16854 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:40.958776 16462 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:59:40.925781 13925 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:40.926895 16871 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:40.927896 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.902477 16833 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250623 01:59:40.758661 16849 consensus_queue.cc:1162] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c243715fe12445019c33bbeafbaea635" responder_term: 11 status { last_received { term: 8 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 10. Current term is 11. Ops: [10.7-10.7]" } } last_received_current_leader { term: 0 index: 0 } }
I20250623 01:59:40.900969 16940 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:40.930670 16853 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:40.931038 16762 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.756106 16888 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.929678 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:40.922184 16823 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:40.938179 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:40.938968 13852 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:40.986943 13810 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:40.953716 13886 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:40.957677 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:40.957180 13809 raft_consensus.cc:2466] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 12.
I20250623 01:59:40.964629 13953 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:40.951478 16866 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:40.935335 16864 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:40.989166 13779 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:40.956180 16841 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:40.965835 16836 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20250623 01:59:40.972060 16853 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.979722 16869 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:40.980059 16840 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:40.981094 16833 consensus_queue.cc:1046] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:40.983083 16888 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:40.970712 16854 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250623 01:59:40.988886 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 50020.6us
I20250623 01:59:40.999225 16839 raft_consensus.cc:3053] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:40.999601 16839 raft_consensus.cc:738] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:41.000355 16839 consensus_queue.cc:260] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.001442 16839 raft_consensus.cc:3058] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:41.002671 16853 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:41.005290 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:41.009130 16840 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:41.010159 16833 raft_consensus.cc:3053] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:41.010594 16833 raft_consensus.cc:738] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:41.011423 16833 consensus_queue.cc:260] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.012825 16833 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:41.012993 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:40.990303 13854 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:40.990475 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52432 (request call id 262) took 9566 ms. Trace:
I20250623 01:59:40.991772 14479 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:40.993038 16666 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:41.014695 16841 raft_consensus.cc:2802] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 01:59:41.015413 16841 raft_consensus.cc:695] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:41.016301 16666 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:41.016822 16666 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:41.016526 16841 consensus_queue.cc:237] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:40.994705 16884 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.021688 13953 raft_consensus.cc:1273] T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:41.023895 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:41.023420 16884 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:40.996459 13961 raft_consensus.cc:604] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:41.025398 13961 raft_consensus.cc:612] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:41.027125 16912 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:41.027210 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:41.027923 13957 raft_consensus.cc:1238] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:41.027710 16912 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.028695 13925 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:41.029680 16912 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:41.029858 13925 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:41.029724 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:41.027953 13962 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:41.027348 13958 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:40.995709 16762 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:41.033107 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:41.034010 13958 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "afad1ff768904b11936d338d33df7998"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
W20250623 01:59:40.962785 16898 consensus_meta.cc:287] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.699s user 0.011s sys 0.003s
I20250623 01:59:40.994138 16849 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:41.034929 16849 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:41.035341 16849 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:41.016988 16886 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:41.036643 16886 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.017922 16800 raft_consensus.cc:2802] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Leader election won for term 12
I20250623 01:59:41.037411 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:41.037709 16800 raft_consensus.cc:695] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:41.038056 13883 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:59:41.038847 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:41.038697 16800 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.040060 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:41.019039 16856 raft_consensus.cc:3053] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:41.041457 16856 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:41.042275 16856 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.042786 13813 raft_consensus.cc:1273] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:41.043531 16856 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:40.968746 16847 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:41.034564 13958 raft_consensus.cc:604] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:41.053485 13958 raft_consensus.cc:612] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:41.024796 13960 raft_consensus.cc:1273] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:41.013057 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:41.054067 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:41.046627 13953 raft_consensus.cc:1273] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250623 01:59:41.047417 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:41.050652 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:41.054657 13886 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:41.051636 13885 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:41.052206 13925 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:41.058073 13806 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:41.059466 13955 raft_consensus.cc:2408] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250623 01:59:41.060796 13809 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:41.062194 13925 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:41.054886 13808 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:41.064426 13888 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 2 FOLLOWER]: Advancing to term 4
I20250623 01:59:41.059388 14478 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:41.013993 13710 rpcz_store.cc:276] 0623 01:59:31.424135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:31.425290 (+ 1155us) service_pool.cc:225] Handling call
0623 01:59:40.990398 (+9565108us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:39.431012 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:39.431586 (+ 574us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:39.431624 (+ 38us) write_op.cc:435] Acquired schema lock
0623 01:59:39.431647 (+ 23us) tablet.cc:660] Decoding operations
0623 01:59:39.432208 (+ 561us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:39.432376 (+ 168us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:39.432403 (+ 27us) tablet.cc:683] Acquiring locks for 3 operations
0623 01:59:39.432779 (+ 376us) tablet.cc:699] Row locks acquired
0623 01:59:39.432802 (+ 23us) write_op.cc:260] PREPARE: finished
0623 01:59:39.432945 (+ 143us) write_op.cc:270] Start()
0623 01:59:39.433056 (+ 111us) write_op.cc:276] Timestamp: P: 1750643979432892 usec, L: 0
0623 01:59:39.433078 (+ 22us) op_driver.cc:348] REPLICATION: starting
0623 01:59:39.512558 (+ 79480us) log.cc:844] Serialized 1598 byte log entry
0623 01:59:40.133930 (+621372us) op_driver.cc:464] REPLICATION: finished
0623 01:59:40.538378 (+404448us) write_op.cc:301] APPLY: starting
0623 01:59:40.549556 (+ 11178us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:40.577195 (+ 27639us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:40.588583 (+ 11388us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:40.591760 (+ 3177us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:40.591955 (+ 195us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0623 01:59:40.592008 (+ 53us) write_op.cc:312] APPLY: finished
0623 01:59:40.592212 (+ 204us) log.cc:844] Serialized 35 byte log entry
0623 01:59:40.592383 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:40.592476 (+ 93us) write_op.cc:454] Released schema lock
0623 01:59:40.592515 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:40.592860 (+ 345us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:40.592888 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1918323,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":368363,"apply.run_cpu_time_us":2128,"apply.run_wall_time_us":64972,"num_ops":3,"prepare.queue_time_us":93765,"prepare.run_cpu_time_us":4752,"prepare.run_wall_time_us":101184,"replication_time_us":566809,"thread_start_us":337720,"threads_started":2}]]}
I20250623 01:59:41.065642 16914 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:41.066601 16914 raft_consensus.cc:491] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:41.066972 16914 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:41.043990 16926 raft_consensus.cc:513] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.082543 13957 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250623 01:59:41.082283 13925 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:41.075685 16840 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:41.084579 13956 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:41.088621 16886 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:41.091109 13957 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:41.091742 13957 raft_consensus.cc:604] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:41.092074 13957 raft_consensus.cc:612] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:41.092533 16823 consensus_queue.cc:1046] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:41.057420 16822 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:41.093582 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:59:41.053000 16853 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:41.099879 16848 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:41.101917 16826 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:59:41.102705 16938 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:41.104475 16826 consensus_queue.cc:1035] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:41.105221 16462 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.106945 16822 raft_consensus.cc:3053] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:41.107329 16822 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:41.107609 13956 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:41.108243 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:41.109336 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:41.109395 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:41.109691 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:41.110307 13807 raft_consensus.cc:2391] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 4: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:41.086398 13925 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:41.086320 13779 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:41.076808 16645 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:41.117493 16836 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:41.118826 16836 consensus_queue.cc:1035] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:41.120682 13925 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:41.119632 16938 raft_consensus.cc:695] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:41.123193 16899 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250623 01:59:41.101477 16944 log.cc:927] Time spent T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.271s user 0.001s sys 0.000s
I20250623 01:59:41.127590 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:41.130725 13925 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:41.133116 13955 raft_consensus.cc:2366] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:59:41.142517 13854 leader_election.cc:400] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:59:41.142966 13854 leader_election.cc:403] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:41.144207 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:41.114566 16866 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:41.147254 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:41.151476 13804 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:41.158499 13888 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:41.124287 16840 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:41.124400 16940 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:41.123605 16938 consensus_queue.cc:237] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.135594 16645 raft_consensus.cc:695] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:41.115728 16848 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20250623 01:59:41.175051 16645 consensus_queue.cc:237] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.176581 16863 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:41.131470 16884 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:41.160015 16851 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 3. Current term is 4.
I20250623 01:59:41.197746 13804 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 9.
I20250623 01:59:41.177928 16937 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:41.201481 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:41.208256 13960 raft_consensus.cc:3058] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:41.165117 16889 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250623 01:59:41.188373 16950 log.cc:927] Time spent T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.334s user 0.002s sys 0.000s
I20250623 01:59:41.200222 14479 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:59:41.211359 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:41.215714 13953 raft_consensus.cc:2391] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 4: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:41.114002 16822 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.129516 16949 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20250623 01:59:41.146932 16886 consensus_queue.cc:1046] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.059s
I20250623 01:59:41.177143 16863 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.205687 16882 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:41.213785 16889 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:41.213889 16462 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:41.236505 13812 raft_consensus.cc:1273] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:41.225549 16804 raft_consensus.cc:2802] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:41.243541 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:41.248983 13885 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:41.254861 16822 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:41.268296 16666 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.240177 16889 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:41.270481 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:41.310588 13886 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:41.270483 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:41.271540 13959 raft_consensus.cc:1273] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20250623 01:59:41.311410 13959 pending_rounds.cc:85] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 4
I20250623 01:59:41.311715 13959 pending_rounds.cc:107] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20250623 01:59:41.311975 13959 raft_consensus.cc:2887] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:41.290189 13812 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250623 01:59:41.290674 16926 leader_election.cc:290] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:41.292235 16863 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:41.292327 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:41.317942 13962 raft_consensus.cc:3058] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:41.213559 16849 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.295408 16804 raft_consensus.cc:695] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [term 9 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:41.305859 13779 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:41.306705 13960 raft_consensus.cc:2466] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 9.
I20250623 01:59:41.307447 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:41.336215 13961 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:41.307677 13925 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:41.238408 16886 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20250623 01:59:41.326586 13954 raft_consensus.cc:1273] T 92909e3375c549b5baa0595e1750fa1f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250623 01:59:41.343238 13779 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:41.354250 16849 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:41.357074 16856 consensus_queue.cc:1035] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:41.358089 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:41.358857 14515 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 9: Already voted for candidate 7d22741cfb9b408684cb82c58241b2f5 in this term.
I20250623 01:59:41.360059 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:41.340373 16926 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20250623 01:59:41.364389 16804 consensus_queue.cc:237] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.366410 16837 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:41.366981 16837 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.368872 16837 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:41.369407 16574 consensus_queue.cc:1035] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:41.373461 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:41.374078 13888 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:41.375326 13925 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:41.343930 16884 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:59:41.342320 16854 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:41.347303 16873 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250623 01:59:41.386797 13886 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:41.350692 16863 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:41.402211 13885 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 10.
I20250623 01:59:41.403900 13962 raft_consensus.cc:2466] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 10.
I20250623 01:59:41.405766 14478 leader_election.cc:304] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:41.387869 16904 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:41.400887 16849 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:41.403525 16863 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:41.406876 16884 raft_consensus.cc:695] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:41.399890 16854 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.420791 16924 raft_consensus.cc:2802] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:41.424240 16904 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:41.431528 16863 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:41.453305 16904 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:41.437104 16914 raft_consensus.cc:513] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.434808 16849 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.508277 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:41.478817 16645 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:41.544164 16954 log.cc:927] Time spent T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
I20250623 01:59:41.509186 16666 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:41.612797 16958 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
W20250623 01:59:41.614078 16956 log.cc:927] Time spent T 0c8353de982c4260a5ecf1445df4ba21 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20250623 01:59:41.546710 16645 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.651340 16645 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:41.599020 16884 consensus_queue.cc:237] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:41.652451 16960 log.cc:927] Time spent T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
I20250623 01:59:41.684870 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:41.689116 13807 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:41.673483 16848 raft_consensus.cc:3053] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Stepping down as leader of term 7
W20250623 01:59:41.692260 16964 log.cc:927] Time spent T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20250623 01:59:41.682332 16836 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:41.673789 16961 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.267s user 0.001s sys 0.000s
I20250623 01:59:41.683669 16854 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:41.706168 13958 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20250623 01:59:41.709486 16836 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.695639 16848 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:41.720113 16849 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:41.710151 16967 log.cc:927] Time spent T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
W20250623 01:59:41.734515 16976 log.cc:927] Time spent T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20250623 01:59:41.735899 16966 log.cc:927] Time spent T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20250623 01:59:41.747985 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:41.748786 13810 raft_consensus.cc:2391] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 10: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:41.748879 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:41.709270 16914 leader_election.cc:290] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:41.749737 13885 raft_consensus.cc:2391] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:41.750993 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:41.753860 13808 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:41.704157 16924 raft_consensus.cc:695] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [term 10 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:41.765082 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:41.766116 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 54739.3us
I20250623 01:59:41.767519 13811 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:41.769548 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:41.770306 13812 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:41.776717 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "272c03e6252d489cbf2fdceb36b05dc1" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:41.777442 13885 raft_consensus.cc:3058] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 7
I20250623 01:59:41.778898 13804 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20250623 01:59:41.783222 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 49315.7us
I20250623 01:59:41.788821 13885 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:41.790570 13808 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:41.805145 13807 raft_consensus.cc:2466] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 7.
I20250623 01:59:41.806480 13852 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:41.777071 16912 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.032s
I20250623 01:59:41.810144 14479 leader_election.cc:304] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters:
I20250623 01:59:41.815310 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:41.815918 13808 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
W20250623 01:59:41.808811 16847 consensus_meta.cc:287] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.746s user 0.009s sys 0.003s
W20250623 01:59:41.817581 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 178305us
I20250623 01:59:41.818146 16973 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20250623 01:59:41.826104 16826 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:41.816243 16800 raft_consensus.cc:2802] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:41.829566 16836 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:41.800158 16848 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.877964 16980 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:41.882382 16889 consensus_meta.cc:287] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.563s user 0.011s sys 0.000s
I20250623 01:59:41.875427 16826 raft_consensus.cc:695] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:41.899902 13926 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:41.888002 16800 raft_consensus.cc:695] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 01:59:41.825996 16924 consensus_queue.cc:237] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:41.886368 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 94994us
I20250623 01:59:41.904785 13926 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:41.913492 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 106614us
I20250623 01:59:41.916260 13808 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20250623 01:59:41.919687 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:41.920331 13959 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:41.922561 13779 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:41.928220 13959 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:41.930083 13959 raft_consensus.cc:1273] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:41.931144 13957 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20250623 01:59:41.931344 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:41.933507 13959 raft_consensus.cc:2391] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:41.893728 16889 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.918185 16822 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250623 01:59:41.935295 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 128902us
I20250623 01:59:41.935648 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:41.936335 13808 raft_consensus.cc:1238] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:41.923204 16823 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:41.893282 16980 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:41.928215 16849 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:41.969714 13955 raft_consensus.cc:2366] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10.
I20250623 01:59:41.970471 13852 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:41.962828 16823 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:41.966513 16822 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:41.938247 16804 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20250623 01:59:41.966341 16849 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:41.987888 16822 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:41.989738 16888 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:41.992205 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 74718.2us
W20250623 01:59:41.994390 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 70331.7us
I20250623 01:59:41.977445 16823 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:41.965675 16937 consensus_queue.cc:1046] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20250623 01:59:41.964692 16836 consensus_queue.cc:1035] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20250623 01:59:41.996243 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:41.999269 13805 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:41.938406 16826 consensus_queue.cc:237] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:41.978291 16988 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:41.968930 16870 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.028s
I20250623 01:59:42.004359 13808 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:42.006491 13805 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:41.993752 16849 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:42.000746 16904 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.015998 16989 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250623 01:59:42.019636 16980 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:41.997895 16854 consensus_queue.cc:1046] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20250623 01:59:42.021412 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.022240 16854 raft_consensus.cc:3053] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:42.022594 16854 raft_consensus.cc:738] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:42.023350 16854 consensus_queue.cc:260] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.024502 16854 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:42.024873 16863 consensus_meta.cc:287] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.564s user 0.007s sys 0.002s
I20250623 01:59:42.025135 16904 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:41.998010 16888 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:42.008839 16988 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:42.033675 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.043525 13805 raft_consensus.cc:2391] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 in current term 10: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:42.035507 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 120) took 2645 ms. Trace:
I20250623 01:59:42.035856 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:42.037987 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:42.042037 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 70343.4us
I20250623 01:59:42.048036 13806 raft_consensus.cc:2391] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 10: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:42.044343 13708 rpcz_store.cc:276] 0623 01:59:39.390533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:39.390683 (+ 150us) service_pool.cc:225] Handling call
0623 01:59:42.035432 (+2644749us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:41.117165 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:41.117734 (+ 569us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:41.117766 (+ 32us) write_op.cc:435] Acquired schema lock
0623 01:59:41.117783 (+ 17us) tablet.cc:660] Decoding operations
0623 01:59:41.118313 (+ 530us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:41.118389 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:41.118415 (+ 26us) tablet.cc:683] Acquiring locks for 3 operations
0623 01:59:41.118734 (+ 319us) tablet.cc:699] Row locks acquired
0623 01:59:41.118751 (+ 17us) write_op.cc:260] PREPARE: finished
0623 01:59:41.118887 (+ 136us) write_op.cc:270] Start()
0623 01:59:41.119045 (+ 158us) write_op.cc:276] Timestamp: P: 1750643981118840 usec, L: 0
0623 01:59:41.119060 (+ 15us) op_driver.cc:348] REPLICATION: starting
0623 01:59:41.162847 (+ 43787us) log.cc:844] Serialized 1372 byte log entry
0623 01:59:41.440696 (+277849us) op_driver.cc:464] REPLICATION: finished
0623 01:59:41.710200 (+269504us) write_op.cc:301] APPLY: starting
0623 01:59:41.714496 (+ 4296us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:41.738109 (+ 23613us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:41.779561 (+ 41452us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:41.812637 (+ 33076us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:41.818502 (+ 5865us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0623 01:59:41.835604 (+ 17102us) write_op.cc:312] APPLY: finished
0623 01:59:41.884752 (+ 49148us) log.cc:844] Serialized 35 byte log entry
0623 01:59:41.889067 (+ 4315us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:41.892250 (+ 3183us) write_op.cc:454] Released schema lock
0623 01:59:41.893158 (+ 908us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:41.928769 (+ 35611us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:41.929681 (+ 912us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":136173,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":258756,"apply.run_cpu_time_us":2839,"apply.run_wall_time_us":208430,"num_ops":3,"prepare.queue_time_us":146858,"prepare.run_cpu_time_us":3179,"prepare.run_wall_time_us":90015,"replication_time_us":303910,"thread_start_us":286924,"threads_started":2}]]}
I20250623 01:59:42.053115 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.054016 13885 raft_consensus.cc:2408] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250623 01:59:41.976900 16800 consensus_queue.cc:237] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:42.062399 13888 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20250623 01:59:42.065714 14479 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:42.067670 13958 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:42.049317 16889 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:42.053542 16863 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:42.048979 16988 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:42.081465 16981 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.369s user 0.004s sys 0.003s
I20250623 01:59:42.083827 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:42.084415 13957 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:42.093164 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:42.093780 13888 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:42.097460 16994 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:42.058687 16926 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:42.099676 16926 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:42.102242 16926 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.103650 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.104401 13808 raft_consensus.cc:2366] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6.
I20250623 01:59:42.105681 14479 leader_election.cc:400] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6.
I20250623 01:59:42.105952 14479 leader_election.cc:403] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:42.106038 16882 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250623 01:59:42.106537 16991 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:42.107410 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:42.107609 16823 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.110124 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.111274 16849 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.112936 16863 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:42.114156 16823 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.115903 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:42.117573 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:42.118430 13885 raft_consensus.cc:2391] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:42.122766 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:42.125248 16863 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.125840 16863 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:42.129057 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.135573 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:42.136125 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 164202us
I20250623 01:59:42.113229 16849 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.143452 16823 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.147711 16988 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.148399 16823 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.149899 16988 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.150689 16823 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.147603 16991 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6.
I20250623 01:59:42.160897 16848 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:42.161052 16822 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.164036 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:42.164577 16888 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.165251 16666 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.165458 16826 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:42.165566 16907 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.167311 16863 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.173949 16994 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250623 01:59:42.178159 16963 log.cc:927] Time spent T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.533s user 0.004s sys 0.000s
I20250623 01:59:42.180516 13808 raft_consensus.cc:1273] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20250623 01:59:42.259878 14507 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:42.185135 16849 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.188136 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.261144 13886 raft_consensus.cc:2391] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:42.193480 16836 raft_consensus.cc:3053] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:42.194628 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:42.264552 13880 raft_consensus.cc:3058] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:42.195902 13887 raft_consensus.cc:1238] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:42.196770 13882 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:59:42.266250 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:42.198840 13759 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 318e5309795f47f3868e0787ddfdd146
I20250623 01:59:42.199182 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:42.202481 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:42.202914 13779 leader_election.cc:400] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10.
I20250623 01:59:42.203797 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:42.269575 13779 leader_election.cc:403] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20250623 01:59:42.205686 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.270206 13954 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:42.208596 13955 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:42.212653 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 59005.9us
I20250623 01:59:42.217680 13962 raft_consensus.cc:1273] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20250623 01:59:42.219410 13957 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:42.166013 16995 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.223295 13888 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:42.232118 13961 raft_consensus.cc:2391] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:42.230435 13709 catalog_manager.cc:5582] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:42.252440 16666 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.258634 16826 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.204020 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.270416 13883 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:42.277575 13709 catalog_manager.cc:5582] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:42.279083 13884 raft_consensus.cc:2391] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
W20250623 01:59:42.278891 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 76102.2us
W20250623 01:59:42.269047 13759 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 318e5309795f47f3868e0787ddfdd146
I20250623 01:59:42.310084 13709 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 11 to 10, leader changed from <none> to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 10 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:42.311892 13779 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:42.312199 13709 catalog_manager.cc:5582] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:42.313704 13709 catalog_manager.cc:5582] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 7, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:42.309542 13776 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.284853 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.293174 16888 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.303144 16996 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.306011 14507 consensus_queue.cc:237] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:42.307703 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:42.317690 13709 catalog_manager.cc:5582] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:42.319376 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.319299 13709 catalog_manager.cc:5582] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:42.319818 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.319964 13805 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:42.320662 13807 raft_consensus.cc:2408] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20250623 01:59:42.321815 13926 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:42.322999 13709 catalog_manager.cc:5582] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:42.324291 13880 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:42.335719 13811 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:42.324658 13709 catalog_manager.cc:5582] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:42.341359 16575 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:42.342073 16840 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:42.343288 16666 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.327453 16888 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.325403 16992 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:42.345254 16888 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.264333 16836 raft_consensus.cc:738] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:42.334358 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:42.313369 16870 consensus_queue.cc:1035] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20250623 01:59:42.316788 16847 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:42.327277 16823 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:42.341813 13709 catalog_manager.cc:5582] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:42.389597 13709 catalog_manager.cc:5582] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:42.390877 13709 catalog_manager.cc:5582] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:42.392410 13709 catalog_manager.cc:5582] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:42.282380 16849 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.310249 16995 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.332015 13953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 2449) took 1078 ms. Trace:
I20250623 01:59:42.350118 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.399871 13811 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:42.351024 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.354678 13852 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
W20250623 01:59:42.403535 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 48986.9us
I20250623 01:59:42.357945 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.404346 13806 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:42.347718 13925 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:42.406777 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 59124.2us
I20250623 01:59:42.307868 16990 consensus_queue.cc:1035] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.081s
I20250623 01:59:42.373422 17001 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:42.380923 16889 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.282472 16863 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10.
I20250623 01:59:42.380432 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.434739 13961 raft_consensus.cc:2391] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:42.380074 13808 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:42.265281 16907 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.455595 16849 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.456408 13961 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20250623 01:59:42.458748 16995 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.427085 16856 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.459784 16856 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.460230 16913 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250623 01:59:42.470152 16822 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.445436 16840 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.472507 16840 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.474346 16840 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.421643 16988 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:42.327288 16996 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.399583 13881 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:42.399575 13953 rpcz_store.cc:276] 0623 01:59:41.254116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:41.257436 (+ 3320us) service_pool.cc:225] Handling call
0623 01:59:41.258572 (+ 1136us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:41.258908 (+ 336us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:41.258965 (+ 57us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:41.259420 (+ 455us) log.cc:844] Serialized 26 byte log entry
0623 01:59:41.412929 (+153509us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:59:41.412966 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:41.413085 (+ 119us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:42.177637 (+764552us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:42.331744 (+154107us) raft_consensus.cc:1683] finished
0623 01:59:42.331769 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:42.331958 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2617,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":5253888,"thread_start_us":152927,"threads_started":1,"wal-append.queue_time_us":355302}
I20250623 01:59:42.410012 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.481727 13806 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:42.482133 16996 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.413303 14515 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:42.409118 13925 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:42.483052 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 74020.8us
I20250623 01:59:42.398105 13709 catalog_manager.cc:5582] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:42.434898 16836 consensus_queue.cc:260] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.428009 16839 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:42.484129 13709 catalog_manager.cc:5582] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 2 to 4, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 4 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:42.485929 13709 catalog_manager.cc:5582] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 8 to 10, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 10 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } }
I20250623 01:59:42.373862 16837 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.388444 16823 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.368834 16826 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.492764 13709 catalog_manager.cc:5582] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:42.396607 16847 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.494490 16839 raft_consensus.cc:2802] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:42.494557 16837 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.499215 16839 raft_consensus.cc:695] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:42.500739 16837 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.435628 17001 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.501368 13962 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:42.502662 17001 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.503249 17001 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.503765 17001 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.504590 16886 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:42.505092 16886 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.506842 16886 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.500238 16839 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:42.495496 16823 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:42.529197 13882 raft_consensus.cc:1273] T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20250623 01:59:42.438760 16889 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.542109 13926 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:42.548243 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 114225us
I20250623 01:59:42.549386 13852 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.550486 13852 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:42.507530 16992 raft_consensus.cc:695] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:42.552838 16992 consensus_queue.cc:237] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.553819 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.554491 13806 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:42.558732 13925 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:42.561968 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.563648 13926 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:42.564855 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.565598 13962 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:42.512792 16988 raft_consensus.cc:695] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:42.562930 13953 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:42.574182 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.574751 13779 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:42.574941 13805 raft_consensus.cc:2408] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250623 01:59:42.572499 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:42.512730 16575 raft_consensus.cc:695] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:42.575505 13813 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:42.591001 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.591554 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.591699 13811 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:42.592365 13880 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:42.592865 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.593214 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.594579 16847 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.568707 16870 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:42.595393 16926 consensus_queue.cc:1035] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250623 01:59:42.575997 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.595880 16870 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:42.596325 13810 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:42.596539 16987 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.576911 16822 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:42.577275 16840 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:42.582227 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.596992 16987 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.597688 13805 raft_consensus.cc:2391] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 8: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:42.598850 16987 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.599225 16900 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.599697 16900 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.573285 16863 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.601118 16988 consensus_queue.cc:237] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.595652 13852 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.601552 16900 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.601955 16863 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.602478 16900 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:42.603087 16900 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.603436 16900 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:42.583114 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.604604 13807 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:42.606462 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.607229 13888 raft_consensus.cc:2408] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250623 01:59:42.589143 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:42.607461 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.607973 13887 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:42.608210 13805 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 5: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:42.610719 13811 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:42.610903 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.617432 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.617761 13882 raft_consensus.cc:1273] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:42.618229 13811 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:42.618556 13808 raft_consensus.cc:1273] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250623 01:59:42.582741 16854 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:42.619764 16854 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.620182 16854 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:42.621855 16840 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:42.622705 16840 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.596351 16870 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:42.615984 13807 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:42.570344 16995 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250623 01:59:42.627938 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 318449us
I20250623 01:59:42.630153 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.631330 13852 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.632283 13807 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:42.634892 13813 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:42.636775 13852 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.582939 16849 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:42.638758 13852 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.643150 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.611974 16822 raft_consensus.cc:695] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:42.641768 13776 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:42.603940 16863 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.647850 16847 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:42.625708 16837 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.628054 16995 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.654881 13852 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.640075 16949 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:42.640616 16849 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.641739 16907 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.575574 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:42.680413 13807 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250623 01:59:42.686348 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 44688.7us
I20250623 01:59:42.672788 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.611456 17001 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.638533 16856 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.642180 16987 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:42.640064 16888 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.673749 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.692487 13852 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:42.690060 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.697834 13888 raft_consensus.cc:2408] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20250623 01:59:42.698009 13953 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:42.695676 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:42.642313 17013 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:42.650286 16845 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:42.711875 13879 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:42.701843 13882 raft_consensus.cc:1273] T 272c03e6252d489cbf2fdceb36b05dc1 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250623 01:59:42.660478 16847 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.715471 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.718165 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:42.720256 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 144733us
I20250623 01:59:42.720793 13962 raft_consensus.cc:2391] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 9: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:42.689549 16995 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:42.729323 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.729960 13957 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:42.731400 17019 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.731889 17019 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.732321 16836 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:42.733529 16889 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.726290 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.699867 16991 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20250623 01:59:42.714578 16888 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.735432 13813 raft_consensus.cc:1273] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:42.697283 16987 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.684755 16904 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:42.693818 16949 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.727792 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.715314 16849 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:42.744758 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.718581 16907 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:42.745261 16996 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:42.757474 16949 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:42.786528 16854 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.788868 16854 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.735296 17012 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:42.689653 16837 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.719372 17013 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.750607 16889 consensus_queue.cc:1035] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:42.752027 16995 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.727962 16847 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:42.714839 16845 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:42.874760 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:42.726220 16989 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.758172 13813 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:42.754487 17010 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:42.771291 13887 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:42.740008 17001 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:42.758231 16904 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.777333 17016 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.777848 16836 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.749256 16888 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:42.792302 13962 raft_consensus.cc:2366] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:42.806520 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.961192 13884 raft_consensus.cc:2391] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 7: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:42.809365 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.745064 16987 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:42.830157 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.754609 13954 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:42.971467 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:42.905086 13926 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:42.931170 13852 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:42.939666 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 127422us
I20250623 01:59:42.813531 17012 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.962055 13960 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:42.967113 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:42.975643 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 70658.9us
W20250623 01:59:42.976469 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 263483us
I20250623 01:59:42.969089 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:42.714130 16575 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:42.961115 13956 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:42.763244 17021 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.187s
I20250623 01:59:42.904033 17014 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:42.904722 17019 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:42.907217 16845 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:42.934052 16989 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:42.812989 17008 log.cc:927] Time spent T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250623 01:59:42.819563 16994 consensus_queue.cc:1035] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.148s
I20250623 01:59:42.973343 13957 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
W20250623 01:59:42.984629 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 248040us
W20250623 01:59:42.988629 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 252955us
I20250623 01:59:42.760150 16822 consensus_queue.cc:237] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.934720 17001 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:42.993049 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:42.993935 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:43.020499 13812 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:42.939373 17030 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:43.036579 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 53288.1us
I20250623 01:59:42.978142 13807 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:42.992106 16888 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:43.039752 13926 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:42.974490 16836 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:43.041616 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 68410.5us
W20250623 01:59:43.043021 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 111952us
I20250623 01:59:42.982805 17012 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:43.044509 16840 consensus_queue.cc:1035] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:43.004627 17019 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:43.124418 17019 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:43.124837 17019 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:59:43.129799 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 85652.7us
I20250623 01:59:43.130745 13926 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:42.972354 17016 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:42.990968 16949 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.034031 17001 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:42.940090 17010 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.156216 13779 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:43.043452 17032 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:43.164000 13812 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:43.180398 13954 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250623 01:59:43.001005 17014 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.170289 17032 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:43.042806 17030 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:43.231117 17025 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20250623 01:59:43.249353 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 68949.1us
W20250623 01:59:43.238250 17006 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250623 01:59:43.240007 16854 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:43.243559 17032 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:43.253827 16995 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:43.313887 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 157766us
I20250623 01:59:43.330772 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:43.343600 13809 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 4.
I20250623 01:59:43.311795 16992 raft_consensus.cc:2747] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250623 01:59:43.384963 16900 consensus_meta.cc:287] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.781s user 0.007s sys 0.000s
I20250623 01:59:43.394171 13955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 682) took 1186 ms. Trace:
I20250623 01:59:43.394722 13955 rpcz_store.cc:276] 0623 01:59:42.207283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.207486 (+ 203us) service_pool.cc:225] Handling call
0623 01:59:42.207853 (+ 367us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:42.208477 (+ 624us) raft_consensus.cc:1532] Early marking committed up to index 9
0623 01:59:42.527294 (+318817us) log.cc:844] Serialized 14 byte log entry
0623 01:59:42.972924 (+445630us) log.cc:844] Serialized 14 byte log entry
0623 01:59:42.988753 (+ 15829us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:43.003163 (+ 14410us) log.cc:844] Serialized 26 byte log entry
0623 01:59:43.003481 (+ 318us) raft_consensus.cc:1643] Marking committed up to 9
0623 01:59:43.003611 (+ 130us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:43.003746 (+ 135us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:43.393848 (+390102us) raft_consensus.cc:1683] finished
0623 01:59:43.393877 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:43.394072 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2939,"prepare.queue_time_us":55626,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":4622,"raft.queue_time_us":58282,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":10587,"spinlock_wait_cycles":12658944,"thread_start_us":319192,"threads_started":2,"wal-append.queue_time_us":229307}
I20250623 01:59:43.325456 16854 raft_consensus.cc:695] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:43.346493 16840 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:43.330240 17053 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:43.387632 17048 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:43.392222 16904 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:43.346166 17019 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.406320 16840 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:59:43.427318 16870 consensus_meta.cc:287] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.798s user 0.005s sys 0.005s
I20250623 01:59:43.412355 16989 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:43.454802 17044 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20250623 01:59:43.467284 17010 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.467934 16870 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.469254 13880 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:43.470381 16949 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.416956 17013 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.447134 16840 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:43.442188 17048 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.490490 16854 consensus_queue.cc:237] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.498342 17014 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.500448 17030 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:43.422375 16836 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.513146 17016 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.514679 16907 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:43.514995 13884 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:43.515245 16907 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:43.517596 16949 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:43.518245 16949 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.520649 16949 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.522810 17019 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:43.506963 16823 consensus_meta.cc:287] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.946s user 0.007s sys 0.003s
I20250623 01:59:43.420843 17053 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.508145 16837 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:43.419413 17034 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:43.403301 16900 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.551204 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.552124 13887 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 11.
W20250623 01:59:43.540608 16848 consensus_meta.cc:287] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 1.303s user 0.012s sys 0.000s
I20250623 01:59:43.559664 13776 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:43.551785 17024 log.cc:927] Time spent T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.384s user 0.002s sys 0.002s
W20250623 01:59:43.593021 16987 consensus_meta.cc:287] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.606s user 0.008s sys 0.003s
I20250623 01:59:43.593509 16987 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.595537 16987 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:43.565268 16990 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.001s sys 0.009s
W20250623 01:59:43.596251 16990 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.806s user 0.001s sys 0.009s
I20250623 01:59:43.597237 17043 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:43.597988 17026 log.cc:927] Time spent T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.503s user 0.000s sys 0.004s
I20250623 01:59:43.597821 17043 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.561480 17048 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:43.539397 17014 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20250623 01:59:43.600150 16870 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:43.561758 17018 log.cc:927] Time spent T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.549s user 0.000s sys 0.006s
I20250623 01:59:43.601614 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 689) took 1138 ms. Trace:
I20250623 01:59:43.601950 13961 rpcz_store.cc:276] 0623 01:59:42.463255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.463401 (+ 146us) service_pool.cc:225] Handling call
0623 01:59:42.463683 (+ 282us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:42.464052 (+ 369us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:42.464083 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:42.464594 (+ 511us) log.cc:844] Serialized 26 byte log entry
0623 01:59:42.713708 (+249114us) raft_consensus.cc:1643] Marking committed up to 8
0623 01:59:42.713762 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:42.713840 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:43.481348 (+767508us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:43.601351 (+120003us) raft_consensus.cc:1683] finished
0623 01:59:43.601373 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:43.601556 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"spinlock_wait_cycles":356132096,"thread_start_us":248702,"threads_started":1,"wal-append.queue_time_us":469654}
I20250623 01:59:43.604689 16840 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:43.575556 17023 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.659s user 0.000s sys 0.006s
W20250623 01:59:43.597443 17027 log.cc:927] Time spent T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.405s user 0.005s sys 0.000s
I20250623 01:59:43.575650 17013 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:43.540182 16823 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.614639 16848 raft_consensus.cc:491] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:43.549715 17001 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:43.616670 16849 consensus_meta.cc:287] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.827s user 0.010s sys 0.002s
I20250623 01:59:43.666409 17001 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.668285 16823 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:43.668871 13953 raft_consensus.cc:2366] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9.
I20250623 01:59:43.619521 17012 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.669363 16849 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.670574 17012 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.671296 16849 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.550750 17034 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:43.676885 17053 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:43.680951 16845 consensus_meta.cc:287] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.672s user 0.006s sys 0.001s
I20250623 01:59:43.684253 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58122 (request call id 1689) took 1097 ms. Trace:
I20250623 01:59:43.684676 13881 rpcz_store.cc:276] 0623 01:59:42.586590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.589433 (+ 2843us) service_pool.cc:225] Handling call
0623 01:59:42.589794 (+ 361us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:42.590189 (+ 395us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:42.590223 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:42.590743 (+ 520us) log.cc:844] Serialized 26 byte log entry
0623 01:59:42.828675 (+237932us) raft_consensus.cc:1643] Marking committed up to 3
0623 01:59:42.828733 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:42.828828 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:43.684045 (+855217us) raft_consensus.cc:1683] finished
0623 01:59:43.684065 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:43.684199 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3056,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":157,"thread_start_us":237501,"threads_started":1,"wal-append.queue_time_us":412056}
I20250623 01:59:43.688277 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:43.689857 17012 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:43.690369 17012 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.691475 17032 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.688163 16991 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250623 01:59:43.620546 17013 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:43.698726 13927 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.027s sys 0.111s
I20250623 01:59:43.677095 17034 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:43.701745 13927 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.580s user 0.027s sys 0.111s
I20250623 01:59:43.704404 17013 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:43.714380 16840 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:43.714957 17032 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:43.696689 16845 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.706616 13959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 690) took 1209 ms. Trace:
I20250623 01:59:43.727931 13959 rpcz_store.cc:276] 0623 01:59:42.497294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.497713 (+ 419us) service_pool.cc:225] Handling call
0623 01:59:42.497990 (+ 277us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:42.498303 (+ 313us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:42.498328 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:42.514016 (+ 15688us) log.cc:844] Serialized 26 byte log entry
0623 01:59:42.873324 (+359308us) raft_consensus.cc:1643] Marking committed up to 8
0623 01:59:42.873365 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:42.873436 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:43.482405 (+608969us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:43.704027 (+221622us) raft_consensus.cc:1683] finished
0623 01:59:43.704052 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:43.706551 (+ 2499us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":387,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":3591,"spinlock_wait_cycles":13618560,"thread_start_us":303326,"threads_started":1,"wal-append.queue_time_us":368862}
W20250623 01:59:43.762259 13926 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.011s sys 0.031s
I20250623 01:59:43.765304 13954 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:43.766793 13954 raft_consensus.cc:604] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:43.765688 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:43.694492 16848 raft_consensus.cc:513] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:43.731408 16847 consensus_meta.cc:287] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.794s user 0.011s sys 0.000s
I20250623 01:59:43.730655 16823 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:43.762781 13926 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.520s user 0.011s sys 0.031s
I20250623 01:59:43.785441 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.730719 16645 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:43.787276 13884 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:43.793124 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:43.793589 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:43.795514 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:43.796792 13954 raft_consensus.cc:2391] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:43.799170 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:43.799923 13813 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:43.801378 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:43.801687 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:43.801977 13805 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:43.802223 13813 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:43.804469 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:43.805044 13810 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:43.807531 17038 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:43.808081 17038 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.808719 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.809589 16845 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.810353 16848 leader_election.cc:290] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:43.810433 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:43.811179 13808 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:43.811532 16907 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.778861 17021 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:43.791877 16822 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:43.812882 17021 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.813114 16822 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.814945 17021 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.796826 13959 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 11.
I20250623 01:59:43.793599 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:43.795686 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:43.820856 13960 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:43.822453 13805 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:43.823940 13813 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:43.776777 17030 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:43.840386 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.775053 17012 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.840716 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:43.842641 13959 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:43.777050 17048 raft_consensus.cc:491] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:43.786763 16840 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:43.785727 17049 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:43.841491 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.850833 13884 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250623 01:59:43.883674 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.791752 16863 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:43.838507 13960 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:43.897382 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:43.898053 13960 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:43.899116 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:43.899780 13960 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:43.900956 13926 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:43.882643 13883 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 01:59:43.902508 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:43.902863 14509 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20250623 01:59:43.841552 16900 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:43.876540 13882 raft_consensus.cc:2391] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:43.835150 13854 leader_election.cc:400] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:43.909473 13854 leader_election.cc:403] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20250623 01:59:43.910243 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 75244us
I20250623 01:59:43.884032 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:43.911315 13887 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:43.911497 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.798622 17034 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:43.851414 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.814981 16822 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:43.894640 13810 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:43.920614 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:43.921162 13810 raft_consensus.cc:3058] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:43.903443 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:43.884893 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:43.922940 13884 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 9
I20250623 01:59:43.883059 17012 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:43.906698 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 54660.6us
I20250623 01:59:43.909039 13804 raft_consensus.cc:1273] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250623 01:59:43.791996 16823 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.912133 13882 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:43.884598 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:43.856554 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:43.913946 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:43.928514 13805 raft_consensus.cc:2391] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:43.929262 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:43.914592 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:43.918337 13886 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:43.786917 16847 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.829388 16645 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:43.935096 13960 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:43.951508 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:43.952438 13960 raft_consensus.cc:2391] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 8: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:43.925711 13883 raft_consensus.cc:2391] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 7: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:43.928247 13955 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:44.005357 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:44.006496 13805 raft_consensus.cc:3058] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 8
I20250623 01:59:43.943113 13810 raft_consensus.cc:2466] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:43.945050 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:43.948526 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:44.027752 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:44.028465 13810 raft_consensus.cc:2366] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7.
I20250623 01:59:44.029485 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:44.030666 13804 raft_consensus.cc:2408] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20250623 01:59:44.033097 13887 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:43.886767 16840 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.053483 13955 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:44.054819 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:44.055591 13955 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:44.064755 13955 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
W20250623 01:59:44.072336 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 140620us
I20250623 01:59:44.073374 13956 raft_consensus.cc:1273] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250623 01:59:44.073770 13956 pending_rounds.cc:85] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 3
W20250623 01:59:44.073598 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 113831us
I20250623 01:59:44.074059 13956 pending_rounds.cc:107] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250623 01:59:44.074419 13956 raft_consensus.cc:2887] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:44.072422 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.077601 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.078493 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:43.887070 17048 raft_consensus.cc:513] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:44.079066 13810 raft_consensus.cc:3058] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 6
I20250623 01:59:44.078732 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.080283 13960 raft_consensus.cc:2408] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20250623 01:59:44.080466 13882 raft_consensus.cc:1238] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250623 01:59:43.904687 16863 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:43.877555 17030 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:44.083691 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 140633us
I20250623 01:59:43.894654 17049 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.098426 13805 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:44.086261 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.103721 13955 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 11: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:44.094568 13810 raft_consensus.cc:2466] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:44.104625 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:44.104651 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:44.109246 13960 raft_consensus.cc:2366] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7.
I20250623 01:59:44.037731 16988 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6.
I20250623 01:59:44.109288 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:43.922861 17034 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:44.114684 13955 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:44.118089 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.078526 16870 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:44.103533 17026 log.cc:927] Time spent T 6ac15d89950f4485a84912743c361f1d P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250623 01:59:43.943488 17012 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.118335 13955 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:43.925043 17061 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.184s
I20250623 01:59:44.106940 16666 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:44.093940 16836 raft_consensus.cc:491] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:44.127086 13854 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.095614 16575 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:44.134269 13955 raft_consensus.cc:604] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:44.139644 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:44.140743 13804 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:44.115962 17034 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:44.140583 13955 raft_consensus.cc:612] T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:44.146576 13962 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:44.083359 16994 consensus_queue.cc:1035] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20250623 01:59:44.151269 13962 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:44.164531 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:44.139101 17038 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:44.153216 16907 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.186453 13854 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.189059 13805 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:44.138274 16870 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.199880 16997 raft_consensus.cc:2802] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:44.200412 13854 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:44.202646 17081 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
I20250623 01:59:44.203438 16840 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:44.156562 16666 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:44.156865 16836 raft_consensus.cc:513] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:44.149982 16575 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:44.290069 17076 log.cc:927] Time spent T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20250623 01:59:44.284709 16863 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.238574 17021 raft_consensus.cc:2802] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:44.245986 17034 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:44.444631 17043 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.292793 13804 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:44.475042 17034 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.194581 17071 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250623 01:59:44.476065 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:44.392285 17023 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.188s user 0.000s sys 0.005s
W20250623 01:59:44.293509 17073 log.cc:927] Time spent T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
W20250623 01:59:44.351522 13959 consensus_meta.cc:287] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.508s user 0.009s sys 0.000s
I20250623 01:59:44.293228 16836 leader_election.cc:290] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:44.506009 13880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 2265) took 1833 ms (client timeout 2143 ms). Trace:
I20250623 01:59:44.477576 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a94c049f124a68ad6678cff7f838ac" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:44.476114 17071 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:44.532753 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:44.480440 13852 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:44.530293 13880 rpcz_store.cc:269] 0623 01:59:42.672448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.672561 (+ 113us) service_pool.cc:225] Handling call
0623 01:59:44.505789 (+1833228us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b54001a3954. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:44.505915 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1779994368}
I20250623 01:59:44.514480 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:44.477274 13959 raft_consensus.cc:2466] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:44.477653 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1637) took 1162 ms. Trace:
I20250623 01:59:44.361246 13884 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:44.476758 13804 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 01:59:44.533910 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:44.534106 14507 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:44.535073 13883 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:44.537462 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:44.538241 13881 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 9: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:44.538993 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:44.539923 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:44.540325 17071 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:44.540500 13881 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:44.535842 14509 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:44.542186 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:44.544129 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 63815.1us
I20250623 01:59:44.535104 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:44.545078 13886 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:44.536918 13882 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:44.547137 13883 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:44.547267 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:44.536432 13812 rpcz_store.cc:276] 0623 01:59:43.315502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:43.320795 (+ 5293us) service_pool.cc:225] Handling call
0623 01:59:43.321207 (+ 412us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:43.321714 (+ 507us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:43.321752 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:43.329484 (+ 7732us) log.cc:844] Serialized 26 byte log entry
0623 01:59:43.790457 (+460973us) raft_consensus.cc:1643] Marking committed up to 2
0623 01:59:43.790512 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:43.790602 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:44.477404 (+686802us) raft_consensus.cc:1683] finished
0623 01:59:44.477426 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:44.477585 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4679,"raft.queue_time_us":49624,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":20894,"thread_start_us":403233,"threads_started":1,"wal-append.queue_time_us":709354}
W20250623 01:59:44.519217 17022 log.cc:927] Time spent T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 1.503s user 0.003s sys 0.008s
I20250623 01:59:44.563385 13806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48572 (request call id 2423) took 2003 ms. Trace:
I20250623 01:59:44.563737 13806 rpcz_store.cc:276] 0623 01:59:42.560326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.566575 (+ 6249us) service_pool.cc:225] Handling call
0623 01:59:42.566913 (+ 338us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:42.567890 (+ 977us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:42.567935 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:42.568418 (+ 483us) log.cc:844] Serialized 26 byte log entry
0623 01:59:42.742030 (+173612us) raft_consensus.cc:1643] Marking committed up to 7
0623 01:59:42.743984 (+ 1954us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:42.744142 (+ 158us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:44.518681 (+1774539us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:44.563120 (+ 44439us) raft_consensus.cc:1683] finished
0623 01:59:44.563140 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:44.563332 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":573,"raft.queue_time_us":734,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":29949,"spinlock_wait_cycles":291770624,"thread_start_us":172892,"threads_started":1,"wal-append.queue_time_us":405613}
I20250623 01:59:44.565768 17071 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.567951 13888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59310 (request call id 712) took 1199 ms. Trace:
I20250623 01:59:44.568349 13888 rpcz_store.cc:276] 0623 01:59:43.368477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:43.369292 (+ 815us) service_pool.cc:225] Handling call
0623 01:59:44.567281 (+1197989us) spinlock_profiling.cc:243] Waited 896 us on lock 0x7b5800002498. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:59:44.567477 (+ 196us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:44.567642 (+ 165us) raft_consensus.cc:1532] Early marking committed up to index 0
0623 01:59:44.567668 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:44.567712 (+ 44us) raft_consensus.cc:1643] Marking committed up to 6
0623 01:59:44.567744 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:44.567795 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:44.567888 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2061696}
I20250623 01:59:44.551384 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:44.552270 13852 leader_election.cc:400] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7.
I20250623 01:59:44.569948 13852 leader_election.cc:403] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20250623 01:59:44.494626 17089 log.cc:927] Time spent T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.285s user 0.000s sys 0.000s
W20250623 01:59:44.301895 17084 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20250623 01:59:44.553282 16992 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:44.553598 16899 raft_consensus.cc:2747] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:44.554904 17068 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:44.300314 17048 leader_election.cc:290] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:44.298336 17014 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:44.557600 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:44.561766 13776 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:44.562244 17013 consensus_meta.cc:287] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.848s user 0.012s sys 0.000s
I20250623 01:59:44.441602 16870 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.292845 17030 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:44.574101 13804 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:44.290319 16645 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:44.291990 16823 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:44.477906 17041 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:44.479091 16997 raft_consensus.cc:695] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:44.610751 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:44.611305 16997 consensus_queue.cc:237] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.612565 17012 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.478634 16575 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.578620 13778 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:44.574947 13852 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.615780 13852 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.616608 13852 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.576313 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:44.617839 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 716929us
W20250623 01:59:44.617926 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 702175us
I20250623 01:59:44.618347 16899 raft_consensus.cc:2802] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:44.618611 13926 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:44.618952 16899 raft_consensus.cc:695] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:44.619791 16899 consensus_queue.cc:237] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.620725 13925 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:44.307401 17049 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:44.576691 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:44.479286 17021 raft_consensus.cc:695] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:44.603052 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:44.603256 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:44.575531 17019 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7.
I20250623 01:59:44.575011 16992 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.575138 17068 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.598691 17095 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20250623 01:59:44.591478 16863 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:44.591177 17013 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.609997 17041 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:44.611326 13804 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:44.620867 16575 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250623 01:59:44.661367 13953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37426 (request call id 2477) took 1937 ms (client timeout 1644 ms). Trace:
W20250623 01:59:44.661828 13953 rpcz_store.cc:269] 0623 01:59:42.715069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.715233 (+ 164us) service_pool.cc:225] Handling call
0623 01:59:44.652708 (+1937475us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b54000bc2d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:44.652856 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2178764672}
I20250623 01:59:44.621588 13805 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
W20250623 01:59:44.594715 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 120902us
I20250623 01:59:44.634003 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:44.658975 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:44.623342 13852 leader_election.cc:304] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
W20250623 01:59:44.660077 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 70535.7us
I20250623 01:59:44.668022 13810 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:44.593521 17014 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:44.674377 13776 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.674845 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:44.630533 16904 raft_consensus.cc:2762] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250623 01:59:44.669665 17105 raft_consensus.cc:491] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:44.670578 17101 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:44.629666 16870 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:44.630750 17049 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
W20250623 01:59:44.673485 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 56153.1us
I20250623 01:59:44.681833 13805 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:44.683707 13854 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:44.714056 13776 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.713483 13852 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.641513 17001 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:44.715408 13807 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 01:59:44.683697 13812 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:44.717015 13881 raft_consensus.cc:2466] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:44.653584 16863 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:44.732095 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 111447us
I20250623 01:59:44.734542 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.728413 13804 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:44.661448 17034 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.736554 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.737653 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.738308 13953 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:44.737689 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de00fb2e24074586bfe02d33fb92e4e6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:44.738745 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 55120.8us
I20250623 01:59:44.739163 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:44.739552 13854 leader_election.cc:400] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7.
I20250623 01:59:44.739799 13960 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:44.740372 13854 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:44.741060 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:44.741824 13959 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:44.741891 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:44.710626 16995 raft_consensus.cc:2747] T d2a94c049f124a68ad6678cff7f838ac P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:44.671885 16575 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:44.782488 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 69121.2us
I20250623 01:59:44.784374 16851 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:44.784794 16851 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:44.785202 16851 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:44.785380 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:44.747123 13956 raft_consensus.cc:2391] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 9: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:44.750782 13962 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6ac15d89950f4485a84912743c361f1d"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "ee041a8325864e788a383b8356afc652"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:44.788976 13962 raft_consensus.cc:604] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Received request to transfer leadership to TS ee041a8325864e788a383b8356afc652
I20250623 01:59:44.789324 13962 raft_consensus.cc:612] T 6ac15d89950f4485a84912743c361f1d P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:44.752692 13959 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:44.682893 17012 raft_consensus.cc:491] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:44.757742 17106 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:44.716068 17019 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250623 01:59:44.761265 17041 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:44.764896 17068 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:44.824205 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 204558us
I20250623 01:59:44.768427 17095 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:44.768864 17071 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.716208 17101 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.709721 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:44.722785 17105 raft_consensus.cc:513] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:44.870407 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 88772us
I20250623 01:59:44.769429 17021 consensus_queue.cc:237] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:44.770288 17074 log.cc:927] Time spent T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250623 01:59:44.771144 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:44.771296 17014 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:44.676178 16840 raft_consensus.cc:491] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:44.772898 16992 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.722127 17020 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:44.774158 17038 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:44.744032 16870 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:59:44.775183 17093 log.cc:927] Time spent T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
I20250623 01:59:44.776331 16886 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:44.737885 16863 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:44.777500 17049 raft_consensus.cc:695] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:44.777907 17034 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20250623 01:59:44.778692 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 45175.5us
I20250623 01:59:44.908250 13925 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.780670 16847 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.821354 17012 raft_consensus.cc:513] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.815352 17106 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:44.822631 17019 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:44.766350 17013 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:44.825111 13959 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:44.922099 13959 raft_consensus.cc:604] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:44.842690 13854 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.796651 13809 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 7 to 9, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:44.783273 16575 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:44.844300 13926 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:44.925910 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 83312.8us
W20250623 01:59:44.926056 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 81857us
I20250623 01:59:44.754634 17001 raft_consensus.cc:695] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:44.790707 16836 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:44.914731 17034 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:44.930375 13926 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:44.933008 13854 leader_election.cc:304] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:44.934999 13854 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:44.935672 13961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34194 (request call id 702) took 1226 ms. Trace:
I20250623 01:59:44.936091 13961 rpcz_store.cc:276] 0623 01:59:43.709019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:43.709592 (+ 573us) service_pool.cc:225] Handling call
0623 01:59:43.778249 (+ 68657us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b5800016e98. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0b02ec1 00007f41e0b02d72 00007f41e97f08f9 00007f41e897bac0 00007f41e8a427bb 00007f41e8a42841 00007f41e8a05860 00007f41e89f0c29 00007f41e89f06d2 00007f41e208a744
0623 01:59:43.778425 (+ 176us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:44.097502 (+319077us) raft_consensus.cc:1532] Early marking committed up to index 10
0623 01:59:44.097802 (+ 300us) log.cc:844] Serialized 14 byte log entry
0623 01:59:44.633568 (+535766us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:44.633659 (+ 91us) raft_consensus.cc:1643] Marking committed up to 10
0623 01:59:44.633719 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:44.935463 (+301744us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b5400244ad4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:44.935506 (+ 43us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:44.935614 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":579005,"prepare.queue_time_us":58296,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":3278,"raft.queue_time_us":561023,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":20210,"spinlock_wait_cycles":741436544}
I20250623 01:59:44.936836 13926 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:44.942029 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.942384 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:44.920356 17106 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:44.901154 17020 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.944458 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:44.945058 13961 raft_consensus.cc:2366] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9.
I20250623 01:59:44.905418 17038 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.945379 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:44.945948 13953 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:44.885200 13960 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:44.890208 13879 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
W20250623 01:59:44.891775 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 97061.5us
I20250623 01:59:44.898955 13957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 2494) took 1106 ms. Trace:
I20250623 01:59:44.929713 17016 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:44.791466 16907 consensus_queue.cc:1046] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.101s
I20250623 01:59:44.901382 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:44.933003 17034 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:44.922680 13959 raft_consensus.cc:612] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:44.922298 17107 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250623 01:59:44.919313 16837 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250623 01:59:44.841666 17095 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:44.952690 13957 rpcz_store.cc:276] 0623 01:59:43.792647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:43.792771 (+ 124us) service_pool.cc:225] Handling call
0623 01:59:43.839787 (+ 47016us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:43.840002 (+ 215us) raft_consensus.cc:1532] Early marking committed up to index 5
0623 01:59:44.129345 (+289343us) log.cc:844] Serialized 14 byte log entry
0623 01:59:44.898532 (+769187us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:44.898589 (+ 57us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:59:44.898647 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:44.898748 (+ 101us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:44.898861 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":529321,"prepare.queue_time_us":278447,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":8781,"raft.queue_time_us":512850,"spinlock_wait_cycles":12718976,"thread_start_us":507327,"threads_started":3,"wal-append.queue_time_us":381645}
I20250623 01:59:44.929816 16823 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:44.823840 17064 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 [term 8 LEADER]: Committing config change with OpId 8.9: config changed from index 7 to 9, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:44.958758 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:44.929647 16836 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:44.960367 13853 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:44.906769 16840 raft_consensus.cc:513] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:44.947047 16856 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:44.938282 16989 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250623 01:59:44.968566 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 145138us
I20250623 01:59:44.953085 16847 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:44.993595 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:44.972958 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 258995us
I20250623 01:59:44.976130 13953 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:44.961503 17016 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:44.940878 16899 raft_consensus.cc:2747] T de00fb2e24074586bfe02d33fb92e4e6 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:44.960266 17013 raft_consensus.cc:491] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:44.963493 17048 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:45.002676 13776 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:45.003007 13953 raft_consensus.cc:2953] T e2928af9e5df4f9eacf59f499ab96dc1 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 7 to 9, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:44.966310 17107 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:45.005790 13926 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:45.016718 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 55437.3us
I20250623 01:59:45.017632 13779 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:45.018784 13779 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:44.979079 17095 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:44.983305 16823 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:44.991917 16989 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:45.021431 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:44.992395 16836 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:45.023416 13779 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:45.062744 13888 raft_consensus.cc:2366] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 11. Current term is 12.
I20250623 01:59:44.952425 16870 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.019464 17107 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:45.005424 16856 raft_consensus.cc:695] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:45.046011 17068 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:45.006474 16847 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.071974 17068 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:45.072448 17068 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:45.073619 16847 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.074201 17105 leader_election.cc:290] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.075145 17038 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:44.960824 17049 consensus_queue.cc:237] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.075443 16863 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.046568 17114 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:45.077632 17114 raft_consensus.cc:695] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:45.078780 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.048803 16992 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:45.079520 16992 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.081442 16992 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.048544 16851 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.051478 17012 leader_election.cc:290] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.006448 17109 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:45.005249 17071 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:45.056066 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.057580 13961 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250623 01:59:45.058420 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.112892 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:45.112979 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.059437 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.070664 16899 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:45.020678 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:44.997789 17001 consensus_queue.cc:237] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:44.978879 16995 consensus_queue.cc:1046] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20250623 01:59:45.063613 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.006628 17016 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
W20250623 01:59:45.064509 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 66294.9us
I20250623 01:59:45.064838 13779 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.037981 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.150319 13852 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:45.150482 13887 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:45.151703 13887 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:45.065681 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.066116 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.067809 13957 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:45.154001 13957 raft_consensus.cc:604] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:45.154356 13957 raft_consensus.cc:612] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:45.156203 17121 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:45.156616 17121 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:45.156927 17121 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:45.157747 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:59:45.069947 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.071524 17104 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:45.078506 17114 consensus_queue.cc:237] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:45.048467 17108 log.cc:927] Time spent T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20250623 01:59:45.081669 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.024673 16904 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:45.024528 16823 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:45.023907 17113 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:45.091094 13776 leader_election.cc:400] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 11. Current term is 12.
I20250623 01:59:45.099305 16851 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.099741 17020 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.054458 17048 raft_consensus.cc:695] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:45.064038 16989 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:45.023268 13854 leader_election.cc:400] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9.
I20250623 01:59:45.119521 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.063427 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.024328 17013 raft_consensus.cc:513] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:45.124478 16863 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.124830 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:45.124989 16899 raft_consensus.cc:695] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:45.229972 16899 consensus_queue.cc:237] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.069792 17101 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.128957 16575 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 01:59:45.143992 17075 log.cc:927] Time spent T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 1.015s user 0.000s sys 0.003s
I20250623 01:59:45.147725 16856 consensus_queue.cc:237] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.147797 13958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37408 (request call id 613) took 2504 ms. Trace:
I20250623 01:59:45.068233 13879 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
W20250623 01:59:45.191445 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 126695us
I20250623 01:59:45.213531 17012 raft_consensus.cc:2747] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:45.206712 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:45.246377 13884 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:45.246515 13810 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:45.207396 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.240684 13958 rpcz_store.cc:276] 0623 01:59:42.643414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.643541 (+ 127us) service_pool.cc:225] Handling call
0623 01:59:45.147228 (+2503687us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400073d54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e89f509e 00007f41e209f8c7 00007f41e209f852 00007f41e209f781 00007f41e209f6fc 00007f41e209e9b2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:45.147716 (+ 488us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2677015808}
I20250623 01:59:45.213583 13959 raft_consensus.cc:2391] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:45.220675 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:45.190376 17109 raft_consensus.cc:695] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:45.225540 17111 log.cc:927] Time spent T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250623 01:59:45.227340 17106 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.233337 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.238993 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.205581 13776 leader_election.cc:403] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20250623 01:59:45.243441 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:45.259456 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 174534us
I20250623 01:59:45.244647 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.212898 13854 leader_election.cc:403] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:45.220680 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.250461 13779 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.212260 16904 raft_consensus.cc:695] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:45.212144 17113 raft_consensus.cc:695] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:45.252166 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:45.261122 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 238186us
I20250623 01:59:45.260102 13776 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.269385 13779 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.159675 17068 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.276362 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:45.281471 13958 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:45.282577 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.282761 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.292032 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.292683 13885 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:45.293991 13810 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 12.
I20250623 01:59:45.266903 17021 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9.
I20250623 01:59:45.305415 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.270097 17014 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:45.309293 17013 leader_election.cc:290] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:45.282133 17105 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:45.310793 16666 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:45.287920 16987 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:45.324712 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:45.256040 17101 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250623 01:59:45.326299 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:45.304417 16840 leader_election.cc:290] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:45.293609 17048 consensus_queue.cc:237] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:45.320328 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 47071.4us
I20250623 01:59:45.336848 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:45.342717 13776 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:45.342705 13853 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:45.319200 17012 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 11. Current term is 12.
I20250623 01:59:45.324019 17050 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:45.344740 16907 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:45.334278 17013 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
W20250623 01:59:45.412159 17090 log.cc:927] Time spent T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20250623 01:59:45.422122 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 152872us
I20250623 01:59:45.444468 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:45.342520 16987 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.490953 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
W20250623 01:59:45.444926 17127 log.cc:927] Time spent T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20250623 01:59:45.445871 17106 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:45.346470 17109 consensus_queue.cc:237] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.346740 17113 consensus_queue.cc:237] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.347412 17105 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:45.426826 17014 consensus_queue.cc:1046] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20250623 01:59:45.347550 16666 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:45.343609 16904 consensus_queue.cc:237] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.452207 16989 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.448989 16575 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:45.322377 16863 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:45.455108 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41526 (request call id 736) took 2010 ms. Trace:
I20250623 01:59:45.512765 13811 rpcz_store.cc:276] 0623 01:59:43.444705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:43.444865 (+ 160us) service_pool.cc:225] Handling call
0623 01:59:43.445204 (+ 339us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:43.445435 (+ 231us) raft_consensus.cc:1532] Early marking committed up to index 9
0623 01:59:43.736758 (+291323us) log.cc:844] Serialized 14 byte log entry
0623 01:59:45.454593 (+1717835us) spinlock_profiling.cc:243] Waited 9.28 ms on lock 0x7b18002858e0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e36eb2ec 00007f41e0b2325c 00007f41e97509b9 00007f41e0b3f7ab 00007f41e0b3da4f 00007f41e0b3fd12 00007f41e36dd04b 00007f41e36e2b53 00007f41e36e306a
0623 01:59:45.454771 (+ 178us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:45.454823 (+ 52us) raft_consensus.cc:1643] Marking committed up to 9
0623 01:59:45.454870 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:45.454935 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:45.455052 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37236,"prepare.queue_time_us":452534,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":6496,"raft.queue_time_us":1595,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":9044,"spinlock_wait_cycles":21350656,"thread_start_us":439372,"threads_started":2,"wal-append.queue_time_us":342471}
I20250623 01:59:45.474126 13776 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:45.444715 17101 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:45.476043 13779 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.518211 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:45.506808 17034 consensus_meta.cc:287] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Time spent flushing consensus metadata: real 0.539s user 0.008s sys 0.000s
I20250623 01:59:45.437000 17095 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:45.527268 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 64319.6us
I20250623 01:59:45.496769 17104 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:45.436878 17121 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.502713 17105 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:45.490104 17013 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.489712 16907 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:45.551270 13958 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:45.530411 16995 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:45.529145 17132 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:45.556543 17114 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:45.488935 17050 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:45.535768 16823 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.560523 17107 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.562996 16907 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:45.567589 13881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58136 (request call id 2275) took 1034 ms. Trace:
I20250623 01:59:45.530740 17101 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.593459 13881 rpcz_store.cc:276] 0623 01:59:44.532809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:44.541119 (+ 8310us) service_pool.cc:225] Handling call
0623 01:59:45.567407 (+1026288us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b540018ef54. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3aa2eda 00007f41e89f1057 00007f41e208ccc4 00007f41e208cc52 00007f41e208cb81 00007f41e208cafc 00007f41e208bdb2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:45.567530 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":386711808}
I20250623 01:59:45.530952 17034 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.537994 17068 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.532663 16863 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:45.554198 13854 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:45.576799 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:59:45.577997 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 661654us
I20250623 01:59:45.604524 13925 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.541066 17105 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:45.584498 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:45.605324 13959 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:45.589751 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:45.607102 13884 raft_consensus.cc:2391] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 9: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:45.568838 16989 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:45.552666 16575 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:45.549530 17016 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.609863 13776 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
W20250623 01:59:45.612443 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 136510us
I20250623 01:59:45.615002 17049 consensus_queue.cc:1046] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:45.541036 17104 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:45.681420 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 69644.4us
I20250623 01:59:45.619402 13962 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 12: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
W20250623 01:59:45.604398 16836 consensus_meta.cc:287] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.522s user 0.005s sys 0.005s
I20250623 01:59:45.590409 17132 raft_consensus.cc:695] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
W20250623 01:59:45.673100 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 63323us
I20250623 01:59:45.675978 13779 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.616097 13925 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.673017 17048 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:45.693876 13807 raft_consensus.cc:1273] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250623 01:59:45.693935 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.698050 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.698395 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.581444 16995 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:45.696066 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 80050.2us
W20250623 01:59:45.699853 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 145745us
I20250623 01:59:45.702795 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.703605 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.703768 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.705832 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e92cd39500a645e79b368d26acb587ed" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.674952 16989 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:45.718981 17110 raft_consensus.cc:491] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:45.719972 16989 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.720144 17110 raft_consensus.cc:513] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:45.721411 17013 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.722193 17110 leader_election.cc:290] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:45.708186 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:45.723084 13961 raft_consensus.cc:2391] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:45.723953 17107 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:45.725355 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:45.723976 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2540092c63cb4d0cbe07ae4018f99f46" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.726011 14507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:45.726948 13961 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250623 01:59:45.729923 16995 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.731168 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:45.693708 17014 raft_consensus.cc:2747] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:45.709460 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.734728 13811 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:45.737534 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:45.738308 13961 raft_consensus.cc:2391] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 9: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:45.739080 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.740334 13961 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 9. Current term is 10. Ops: []
I20250623 01:59:45.740772 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:45.741345 13886 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:45.742147 16870 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:45.722448 16997 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:45.743930 16997 raft_consensus.cc:695] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:45.744992 16997 consensus_queue.cc:237] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.748710 13954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37426 (request call id 2499) took 1945 ms. Trace:
I20250623 01:59:45.749121 13954 rpcz_store.cc:276] 0623 01:59:43.803660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:43.804368 (+ 708us) service_pool.cc:225] Handling call
0623 01:59:43.804649 (+ 281us) raft_consensus.cc:1469] Updating replica for 0 ops
0623 01:59:43.819265 (+ 14616us) raft_consensus.cc:1532] Early marking committed up to index 5
0623 01:59:44.029251 (+209986us) log.cc:844] Serialized 14 byte log entry
0623 01:59:44.759294 (+730043us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b4800211a58. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e8956871 00007f41e894da17 00007f41e894d8aa 00007f41e8a4297f 00007f41e8a42647 00007f41e8a56c6c 00007f41e8a56c18 00007f41e8a56b98 00007f41e8a56b53 00007f41e8a55dfc 00007f41e79dc736
0623 01:59:45.007903 (+248609us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0623 01:59:45.007958 (+ 55us) raft_consensus.cc:1643] Marking committed up to 5
0623 01:59:45.008002 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:45.748486 (+740484us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b54000bc7d4. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a9f9f1 00007f41e3a9e42e 00007f41e89f0719 00007f41e208a744 00007f41e208a6d2 00007f41e208a601 00007f41e208a57c 00007f41e2089832 00007f41e1ea1dcc 00007f41e1ea1206
0623 01:59:45.748534 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:45.748655 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":552661,"prepare.queue_time_us":219591,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":560260,"raft.run_cpu_time_us":2292,"raft.run_wall_time_us":87494,"spinlock_wait_cycles":119737472,"thread_start_us":584881,"threads_started":3,"wal-append.queue_time_us":134507,"wal-append.run_cpu_time_us":10845,"wal-append.run_wall_time_us":1420067}
I20250623 01:59:45.750651 16823 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:45.751266 17013 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:45.752318 17104 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.753249 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.711351 13926 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:45.754807 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 43569.6us
I20250623 01:59:45.757565 16987 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.680449 17030 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:45.758338 17030 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.760344 17030 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:45.760862 17030 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:45.726056 13957 raft_consensus.cc:2366] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10.
I20250623 01:59:45.762902 17121 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.680686 16863 raft_consensus.cc:2747] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:45.710532 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.709690 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c8353de982c4260a5ecf1445df4ba21" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.708932 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.597456 17114 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:45.680652 16575 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:45.717196 17131 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.263s user 0.001s sys 0.000s
I20250623 01:59:45.718940 17016 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.719091 17143 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:45.694343 16836 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.719543 17133 raft_consensus.cc:3053] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:45.722338 17034 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.751909 17013 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.767256 13812 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 10. Current term is 11. Ops: []
I20250623 01:59:45.774703 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:45.724692 17109 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250623 01:59:45.768559 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.686636 17068 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20250623 01:59:45.708987 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:45.713757 17049 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250623 01:59:45.716161 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829478b3e1dc4a95a122fcd8be3d2c77" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:45.788372 13886 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 11.
I20250623 01:59:45.796541 13805 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:45.715082 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb4438d5ce9e41f3a72f00de14912cff" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.771353 17041 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:45.731185 17110 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:45.765645 17101 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.782090 16870 consensus_queue.cc:1046] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20250623 01:59:45.791215 17133 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:45.817924 13953 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250623 01:59:45.814234 16870 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Stepping down as leader of term 9
I20250623 01:59:45.816006 17110 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.827015 13805 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:45.821653 16870 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:45.795341 17143 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.820552 17095 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:45.845643 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 158119us
I20250623 01:59:45.812465 17041 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.848439 13852 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:45.862682 13852 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:45.862607 17110 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.864929 17123 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:45.819037 17132 consensus_queue.cc:237] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.864063 17064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36030 (request call id 281) took 3560 ms. Trace:
I20250623 01:59:45.865978 16870 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.911844 17050 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:45.912467 16870 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:45.846210 17133 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:45.913669 17141 log.cc:927] Time spent T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
W20250623 01:59:45.915728 17146 log.cc:927] Time spent T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20250623 01:59:45.916642 17114 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:45.916685 17143 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.904680 17064 rpcz_store.cc:276] 0623 01:59:42.303239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.303442 (+ 203us) service_pool.cc:225] Handling call
0623 01:59:42.327138 (+ 23696us) log.cc:844] Serialized 487 byte log entry
0623 01:59:42.792741 (+465603us) spinlock_profiling.cc:243] Waited 7.01 ms on lock 0x7b1800346b40. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b404f2 00007f41e0b428c1 00007f41e0b42879 00007f41e0b42809 00007f41e0b427d1 00007f41e0b41acd 00007f41e9b9b124 00007f41e9b9af59 00007f41e0b23646
0623 01:59:43.384936 (+592195us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b1800346b40. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e97d7af1 00007f41e97e4441 00007f41e0b404f2 00007f41e0b428c1 00007f41e0b42879 00007f41e0b42809 00007f41e0b427d1 00007f41e0b41acd 00007f41e9b9b124 00007f41e9b9af59 00007f41e0b23646
0623 01:59:43.779005 (+394069us) spinlock_profiling.cc:243] Waited 58.1 ms on lock 0x7b540055e9a0. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23481 00007f41e3a36ff6 00007f41e3a3096a 00007f41e3a2fbd4 00007f41e3a18372 00007f41e3a1d781 00007f41e3a1d709 00007f41e3a1d699 00007f41e3a1d661 00007f41e3a1c82d 00007f41e9b9b124
0623 01:59:45.863945 (+2084940us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17170,"raft.queue_time_us":8987,"raft.run_cpu_time_us":4631,"raft.run_wall_time_us":1081353,"spinlock_wait_cycles":157573120,"thread_start_us":189194,"threads_started":1,"wal-append.queue_time_us":251192,"wal-append.run_cpu_time_us":1558,"wal-append.run_wall_time_us":1001568}
I20250623 01:59:45.919020 17105 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:45.920356 17041 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.920514 17105 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.916760 17133 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:45.959491 13810 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
W20250623 01:59:45.961982 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 220272us
W20250623 01:59:45.968336 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 105628us
I20250623 01:59:45.977509 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:45.978096 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.979128 13879 raft_consensus.cc:2366] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9.
I20250623 01:59:45.979431 13885 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:45.978130 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.980677 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:45.983475 13881 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 4.
I20250623 01:59:45.985687 13852 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:45.976359 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:45.986384 13776 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:45.992376 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:45.992462 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:45.992759 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:45.998180 13756 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e2928af9e5df4f9eacf59f499ab96dc1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 10)
I20250623 01:59:46.013377 13955 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:46.015751 13926 leader_election.cc:304] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:46.021936 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:46.027546 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:46.028761 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:45.905865 16836 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:45.908412 16575 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.031452 16666 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:46.032671 16840 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:45.915463 17013 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.000119 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.066872 16904 raft_consensus.cc:2747] T e92cd39500a645e79b368d26acb587ed P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:46.082234 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:46.159062 13927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 77641.2us
W20250623 01:59:46.164654 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 149239us
I20250623 01:59:46.166994 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.168116 13879 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:46.168483 17143 raft_consensus.cc:491] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:46.165518 17133 raft_consensus.cc:491] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:46.169490 17143 raft_consensus.cc:513] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:46.170322 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.171820 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.172750 13884 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:46.174657 13776 leader_election.cc:400] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9.
I20250623 01:59:46.175173 13776 leader_election.cc:403] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20250623 01:59:46.169870 17133 raft_consensus.cc:513] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.177835 17143 leader_election.cc:290] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:46.180239 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 194880us
I20250623 01:59:46.180466 17120 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election won for term 12
I20250623 01:59:46.189958 13881 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:46.194280 13807 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:46.197464 17132 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 01:59:46.198146 17132 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:46.199725 17132 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.206159 13807 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:46.187227 17105 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:46.210050 17105 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.217551 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:46.212430 17105 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.190886 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.221686 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.227003 13883 raft_consensus.cc:2391] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 7: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:46.216027 17120 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:46.219978 17113 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9.
I20250623 01:59:46.236624 13925 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:46.237250 17122 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.024s
I20250623 01:59:46.239261 17065 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250623 01:59:46.245757 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 56087us
I20250623 01:59:46.247205 13776 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.249688 17105 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:46.253108 17114 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:46.253649 17114 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:46.254066 17114 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:46.253662 17105 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:46.248195 17113 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:46.243961 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:46.241789 17065 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:46.244589 13779 leader_election.cc:304] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:46.263402 17013 raft_consensus.cc:2747] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:46.235720 17133 leader_election.cc:290] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:46.264070 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 87479.3us
I20250623 01:59:46.264103 17113 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:59:46.270159 17125 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.939s user 0.005s sys 0.000s
I20250623 01:59:46.264346 17065 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:46.270280 17113 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:46.269968 17106 raft_consensus.cc:3053] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:46.269630 17049 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:46.283541 17120 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.273679 17106 raft_consensus.cc:738] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:46.302057 13959 raft_consensus.cc:1273] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20250623 01:59:46.304045 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:46.306900 17106 consensus_queue.cc:260] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.308449 17106 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:46.313041 17123 consensus_queue.cc:1035] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:46.321220 13776 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.329993 17052 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:46.332141 17052 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:46.334406 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.335737 13809 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:46.336539 17052 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.337129 16886 raft_consensus.cc:3053] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:46.337749 16886 raft_consensus.cc:738] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:46.339308 16886 consensus_queue.cc:260] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.340636 16886 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:46.342461 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.342981 13806 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250623 01:59:46.344102 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:46.344911 13806 raft_consensus.cc:2391] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 9: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:46.345927 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.346077 13776 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.346679 17105 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:46.347159 17013 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:46.347654 17013 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:46.348079 17013 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:46.349354 13806 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250623 01:59:46.346019 17012 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:46.354506 17012 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:46.354979 17012 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:46.355818 13804 raft_consensus.cc:1238] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:46.345155 17114 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.358177 17143 raft_consensus.cc:491] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:46.358608 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.358956 17143 raft_consensus.cc:513] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.360085 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.360561 16992 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:46.358187 13886 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:46.361452 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.361179 17143 leader_election.cc:290] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.363294 13779 leader_election.cc:400] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10.
W20250623 01:59:46.363741 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 139759us
I20250623 01:59:46.363945 13779 leader_election.cc:403] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20250623 01:59:46.365033 13926 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:46.365371 17133 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10.
I20250623 01:59:46.365489 13779 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:46.366662 17049 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:46.367153 17049 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:46.367437 13779 leader_election.cc:304] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:46.367635 17049 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:46.370154 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.363040 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:46.347429 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.376076 17113 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.361160 16992 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.372304 17133 raft_consensus.cc:2747] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:46.386951 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.388099 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.389079 13957 raft_consensus.cc:1273] T 5d1d49d7480d4088be0d5ed6baae9489 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:46.393119 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.394232 13805 raft_consensus.cc:1273] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250623 01:59:46.396117 16989 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:46.401228 13886 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:46.363070 17065 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.407632 17143 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:46.409602 17143 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:46.410387 13959 raft_consensus.cc:1273] T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:46.411424 13778 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:46.412557 17095 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:46.414698 13812 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:46.414487 17012 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.417120 13810 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:46.417954 13812 raft_consensus.cc:1273] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:46.419401 13812 raft_consensus.cc:1273] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250623 01:59:46.419329 17121 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:46.420542 13807 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:46.422672 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.423288 13886 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:46.424716 16840 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:46.425366 16840 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.425966 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f12346da5a14097bbfa150cb8d1d907" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.427594 16840 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:46.429075 16904 raft_consensus.cc:491] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:46.429718 16904 raft_consensus.cc:513] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.432006 16904 leader_election.cc:290] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:46.432106 14515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36030 (request call id 283) took 1173 ms. Trace:
I20250623 01:59:46.415788 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:46.435176 13804 raft_consensus.cc:2391] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
W20250623 01:59:46.427695 13852 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:46.441071 13882 raft_consensus.cc:1238] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20250623 01:59:46.380695 17013 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.441507 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.441901 13884 raft_consensus.cc:1238] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:46.442559 13888 raft_consensus.cc:1273] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250623 01:59:46.442791 17105 raft_consensus.cc:2747] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:46.444841 14479 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:46.432613 14515 rpcz_store.cc:276] 0623 01:59:45.258911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:45.259116 (+ 205us) service_pool.cc:225] Handling call
0623 01:59:46.431251 (+1172135us) spinlock_profiling.cc:243] Waited 759 ms on lock 0x7b54006f0754. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e3a23a70 00007f41e3aa7b1a 00007f41e89f509e 00007f41e209f8c7 00007f41e209f852 00007f41e209f781 00007f41e209f6fc 00007f41e209e9b2 00007f41e1ea1dcc 00007f41e1ea1206 00007f41e1ea0baf
0623 01:59:46.431861 (+ 610us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1746768640}
I20250623 01:59:46.461580 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:46.465442 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 79039.9us
I20250623 01:59:46.467952 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.474761 13888 raft_consensus.cc:2391] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 9: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:46.479302 13805 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20250623 01:59:46.482437 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:46.482518 13953 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:46.487599 13959 raft_consensus.cc:1273] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:46.488883 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.495669 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.497756 13959 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:46.504208 17105 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:46.505084 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 137810us
I20250623 01:59:46.505004 17105 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.506036 13779 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:46.506984 13888 raft_consensus.cc:1238] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:46.507469 13779 leader_election.cc:304] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:46.504367 13880 raft_consensus.cc:1273] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:46.467289 17114 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.510324 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:46.511266 13811 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 12: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:46.513084 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.513847 13884 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:46.471079 16886 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250623 01:59:46.523897 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:46.525630 13811 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:45.476303 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:46.529451 17014 raft_consensus.cc:2747] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:46.530279 17068 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:46.534142 16992 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.534494 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.534809 17012 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.535004 17020 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250623 01:59:46.535576 17042 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.018s
I20250623 01:59:46.511828 17013 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.538035 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.538664 13887 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:46.542968 13886 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:46.526386 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.545356 13776 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.545531 13881 raft_consensus.cc:1238] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:46.548039 13888 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:46.541841 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.548532 13884 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:46.554256 13708 catalog_manager.cc:5582] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:46.550599 17013 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:46.561420 13926 leader_election.cc:304] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:46.562374 16840 consensus_queue.cc:1046] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:46.562834 13926 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:46.563390 16904 raft_consensus.cc:3053] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:46.563846 16989 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:46.563962 16904 raft_consensus.cc:738] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:46.564594 16989 raft_consensus.cc:695] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 9 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:46.563321 16840 raft_consensus.cc:2747] T 829478b3e1dc4a95a122fcd8be3d2c77 P c243715fe12445019c33bbeafbaea635 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:46.516374 17049 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.567449 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.589987 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.571542 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.567967 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.592020 17113 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.592218 13888 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:46.592831 13882 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:46.595106 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.596089 13883 raft_consensus.cc:2408] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20250623 01:59:46.556854 13708 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 10 to 12, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 12 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:46.570513 13956 raft_consensus.cc:1238] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:46.597378 13887 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:46.567552 13776 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.600204 17065 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:46.585922 17105 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.579012 17049 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:46.603853 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:46.563860 13926 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:46.584715 17013 raft_consensus.cc:695] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:46.614017 17013 consensus_queue.cc:237] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.605155 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.569916 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:46.640827 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 91749.5us
W20250623 01:59:46.630111 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 62633.7us
I20250623 01:59:46.646713 13776 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.565687 16989 consensus_queue.cc:237] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.565268 16904 consensus_queue.cc:260] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:46.777733 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 44399.6us
I20250623 01:59:46.677376 13887 raft_consensus.cc:1238] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250623 01:59:46.749434 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.655879 13708 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 3 to 4, leader changed from 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 4 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:46.730732 13880 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250623 01:59:46.804298 13880 pending_rounds.cc:85] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Aborting all ops after (but not including) 3
I20250623 01:59:46.804818 13880 pending_rounds.cc:107] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250623 01:59:46.805260 13880 raft_consensus.cc:2887] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250623 01:59:46.806766 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 243027us
I20250623 01:59:46.806903 16840 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:46.807585 16840 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:46.808077 13880 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:46.732558 13882 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:46.808660 16840 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.767095 13884 raft_consensus.cc:1238] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:46.795172 17001 consensus_queue.cc:1046] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:46.811100 17001 raft_consensus.cc:3053] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:46.811452 17001 raft_consensus.cc:738] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:46.812420 17001 consensus_queue.cc:260] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:46.813266 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 83290.7us
I20250623 01:59:46.813625 17014 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:46.814219 17001 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:46.814191 17014 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.795401 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.816181 17049 consensus_queue.cc:1046] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:46.817605 13807 raft_consensus.cc:1238] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 9. Current term is 10. Ops: []
I20250623 01:59:46.817605 13804 raft_consensus.cc:1238] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:46.819519 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:46.819705 13854 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:46.820634 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:46.821750 17014 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:46.819782 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.824434 13779 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:46.825618 13953 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250623 01:59:46.827013 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.827291 17143 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:46.828639 13779 leader_election.cc:304] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:46.829188 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:46.829424 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.829499 17014 raft_consensus.cc:2747] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250623 01:59:46.670768 13881 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:46.757885 13888 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:46.643339 17105 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election won for term 7
W20250623 01:59:46.794131 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:46.818333 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:46.831451 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.831233 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.833098 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.835515 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.837522 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:46.838986 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.825169 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:46.839212 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92909e3375c549b5baa0595e1750fa1f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.839111 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d0bf37b0354956a6d195c5259a3d19" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.840359 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff8dea72f9f9447abaa5fa803f5e84e8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.840839 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d1d49d7480d4088be0d5ed6baae9489" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.842953 17109 consensus_queue.cc:1046] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:46.831281 17105 raft_consensus.cc:695] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:46.699474 13811 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:46.672305 13806 raft_consensus.cc:1273] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:46.845124 17105 consensus_queue.cc:237] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.656095 13776 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.795439 13708 catalog_manager.cc:5582] T 6ac15d89950f4485a84912743c361f1d P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 8 to 10, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 10 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:46.780344 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.850967 13886 raft_consensus.cc:2408] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
W20250623 01:59:46.850198 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 194163us
I20250623 01:59:46.862375 13887 raft_consensus.cc:1238] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 9. Current term is 10. Ops: []
I20250623 01:59:46.658766 17113 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:46.871750 13776 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:46.875455 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:46.875813 13886 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250623 01:59:46.879256 17065 raft_consensus.cc:3053] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:46.879889 17065 raft_consensus.cc:738] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:46.868726 13708 catalog_manager.cc:5582] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 5 to 7, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 7 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:46.859194 17014 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:46.892688 13708 catalog_manager.cc:5582] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 7 to 9, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 9 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:46.904139 13954 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:46.873457 17113 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:46.884037 17013 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election won for term 10
W20250623 01:59:46.913939 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 85384.3us
I20250623 01:59:46.881152 17106 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
W20250623 01:59:46.930071 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:46.941795 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 56725.5us
I20250623 01:59:46.912497 17113 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:46.909065 17014 raft_consensus.cc:695] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:46.959443 13886 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20250623 01:59:46.965819 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:46.881224 17065 consensus_queue.cc:260] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:46.976222 16870 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:46.985551 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:46.986531 13957 raft_consensus.cc:2391] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 10: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:46.946790 17013 raft_consensus.cc:695] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:46.953408 17133 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250623 01:59:46.992851 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:46.995607 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:46.995924 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.029507 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:47.046471 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.072197 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 257) took 7688 ms. Trace:
I20250623 01:59:46.999235 16870 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.088724 13958 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:47.023916 17049 consensus_queue.cc:1046] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20250623 01:59:47.088235 17050 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
W20250623 01:59:47.097460 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 182256us
I20250623 01:59:47.083165 17021 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:47.129469 17107 consensus_queue.cc:1046] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.043s
I20250623 01:59:47.132174 16883 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:47.133116 13852 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:47.134410 17071 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:47.136343 17122 consensus_queue.cc:1046] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20250623 01:59:47.145025 17071 raft_consensus.cc:3053] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:47.145529 17071 raft_consensus.cc:738] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:47.146528 17071 consensus_queue.cc:260] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:47.148003 17071 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:47.148223 16870 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.136492 13852 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:47.166848 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 70190.9us
I20250623 01:59:47.086313 17014 consensus_queue.cc:237] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.170852 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.172469 13962 raft_consensus.cc:2408] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250623 01:59:47.192211 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 243490us
W20250623 01:59:47.205080 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:47.205616 13926 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
W20250623 01:59:47.215297 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 63871.7us
I20250623 01:59:47.113955 17013 consensus_queue.cc:237] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.214955 16870 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:47.224944 17105 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:47.259588 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 123189us
I20250623 01:59:47.184940 17021 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:47.282706 13888 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:47.295578 13852 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:47.265152 17105 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:47.293365 17173 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20250623 01:59:47.283317 16870 consensus_queue.cc:1035] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20250623 01:59:47.311792 17021 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:47.312151 17174 log.cc:927] Time spent T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20250623 01:59:47.077373 13709 rpcz_store.cc:276] 0623 01:59:39.217760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:39.217939 (+ 179us) service_pool.cc:225] Handling call
0623 01:59:46.894799 (+7676860us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:43.554529 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:43.555271 (+ 742us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:43.555321 (+ 50us) write_op.cc:435] Acquired schema lock
0623 01:59:43.555345 (+ 24us) tablet.cc:660] Decoding operations
0623 01:59:43.563955 (+ 8610us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:43.578007 (+ 14052us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:43.589552 (+ 11545us) tablet.cc:683] Acquiring locks for 17 operations
0623 01:59:43.708818 (+119266us) tablet.cc:699] Row locks acquired
0623 01:59:43.708840 (+ 22us) write_op.cc:260] PREPARE: finished
0623 01:59:43.708996 (+ 156us) write_op.cc:270] Start()
0623 01:59:43.709117 (+ 121us) write_op.cc:276] Timestamp: P: 1750643983708944 usec, L: 0
0623 01:59:43.709140 (+ 23us) op_driver.cc:348] REPLICATION: starting
0623 01:59:43.713141 (+ 4001us) log.cc:844] Serialized 6756 byte log entry
0623 01:59:44.505539 (+792398us) op_driver.cc:464] REPLICATION: finished
0623 01:59:44.712356 (+206817us) write_op.cc:301] APPLY: starting
0623 01:59:44.721628 (+ 9272us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:44.756699 (+ 35071us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:44.756830 (+ 131us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:44.759283 (+ 2453us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:44.759687 (+ 404us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0623 01:59:44.759733 (+ 46us) write_op.cc:312] APPLY: finished
0623 01:59:44.760042 (+ 309us) log.cc:844] Serialized 119 byte log entry
0623 01:59:44.761090 (+ 1048us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:44.761333 (+ 243us) write_op.cc:454] Released schema lock
0623 01:59:44.761408 (+ 75us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:44.761803 (+ 395us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:44.761828 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":37366,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":155897,"apply.run_cpu_time_us":4945,"apply.run_wall_time_us":71860,"num_ops":17,"prepare.queue_time_us":877966,"prepare.run_cpu_time_us":27801,"prepare.run_wall_time_us":325078,"raft.queue_time_us":302112,"raft.run_cpu_time_us":10146,"raft.run_wall_time_us":123899,"replication_time_us":796245,"thread_start_us":720606,"threads_started":4,"wal-append.queue_time_us":480669,"wal-append.run_cpu_time_us":23533,"wal-append.run_wall_time_us":1561456}]]}
W20250623 01:59:47.313063 17176 log.cc:927] Time spent T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20250623 01:59:47.315943 17133 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:47.346531 16997 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:47.346408 17133 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:47.348299 17143 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:47.348495 17105 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.349058 17105 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:47.349426 17133 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:47.350385 17184 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250623 01:59:47.348951 17143 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.288080 17183 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20250623 01:59:47.346928 16992 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:47.347011 16997 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:59:47.364341 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 262905us
I20250623 01:59:47.405522 16992 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.366832 16997 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
W20250623 01:59:47.408170 17179 log.cc:927] Time spent T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
W20250623 01:59:47.416625 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 164803us
W20250623 01:59:47.418751 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 113057us
W20250623 01:59:47.427685 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 112023us
I20250623 01:59:47.417479 17188 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:47.435452 13697 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 15e1fc4335c84e18ba2e5c6ea55fed7a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20250623 01:59:47.440464 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 252426us
I20250623 01:59:47.439126 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.448583 16988 consensus_queue.cc:1035] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20250623 01:59:47.449092 16870 consensus_queue.cc:1035] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:47.450937 16870 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:47.451916 16988 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:47.452602 17188 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.459035 17188 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:47.454743 17192 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:47.460006 17192 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:47.460474 17192 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:47.461760 16988 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:47.462931 17178 log.cc:927] Time spent T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.331s user 0.004s sys 0.000s
I20250623 01:59:47.465061 17143 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.465766 17106 raft_consensus.cc:3053] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Stepping down as leader of term 5
I20250623 01:59:47.466169 17106 raft_consensus.cc:738] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:47.467463 17065 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:47.468086 17193 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:47.468649 17193 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.474779 16988 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:47.477733 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:47.431505 16989 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.022s sys 0.004s
W20250623 01:59:47.486398 16989 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s user 0.022s sys 0.005s
W20250623 01:59:47.491034 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 359416us
I20250623 01:59:47.510030 16904 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:47.510120 17193 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.516744 17049 raft_consensus.cc:3053] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:47.517336 17049 raft_consensus.cc:738] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:47.517736 16992 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.518357 17049 consensus_queue.cc:260] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.519734 17049 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:59:47.520550 17001 consensus_meta.cc:287] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.706s user 0.013s sys 0.000s
I20250623 01:59:47.434033 17113 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.502276 16988 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:47.553001 17014 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:47.554790 17014 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:47.555877 17192 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.560129 17014 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:47.560554 13958 raft_consensus.cc:1238] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:47.561829 17192 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:47.562683 13926 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:47.571843 17175 log.cc:927] Time spent T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.345s user 0.003s sys 0.000s
I20250623 01:59:47.574024 17106 consensus_queue.cc:260] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:47.593129 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 163784us
W20250623 01:59:47.605638 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 303893us
I20250623 01:59:47.622220 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:47.622531 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 59945us
I20250623 01:59:47.623319 13926 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:47.624627 13926 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:47.624704 17204 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:47.625454 17204 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.620167 17107 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:47.630339 17107 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.633880 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:47.635135 17191 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250623 01:59:47.635883 17191 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:47.636462 17113 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.619665 13852 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 01:59:47.638797 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 44997.1us
I20250623 01:59:47.639905 13852 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:47.640625 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:47.644425 13926 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:47.608372 17192 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:47.559947 16997 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.587410 16988 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.641299 13852 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:47.653990 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:47.614553 13779 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.87.3:33369 timed out after 1.636s (SENT) [suppressed 4 similar messages]
I20250623 01:59:47.660439 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.618253 17106 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:47.641511 17021 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:47.677765 17020 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.016s
I20250623 01:59:47.679579 17071 consensus_queue.cc:1046] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20250623 01:59:47.695227 17203 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:47.707984 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:47.688999 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:47.701920 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 57609.3us
I20250623 01:59:47.653792 17107 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:47.708305 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:47.660082 17042 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:47.708701 13884 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:47.712829 13883 raft_consensus.cc:2391] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:47.709543 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.687443 17191 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:47.714882 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:47.717869 13961 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20250623 01:59:47.718242 13961 pending_rounds.cc:85] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 5
I20250623 01:59:47.718461 13961 pending_rounds.cc:107] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20250623 01:59:47.718657 13961 raft_consensus.cc:2887] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:47.722805 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:47.723294 13883 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:47.721660 13776 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
W20250623 01:59:47.724473 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 83239.9us
I20250623 01:59:47.724524 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:47.723604 13954 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:47.725539 13887 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:47.727042 13885 raft_consensus.cc:2391] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:47.730443 13926 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:47.628176 17204 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.721127 17050 consensus_queue.cc:1046] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20250623 01:59:47.731925 17133 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:47.747792 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.748353 13955 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250623 01:59:47.748667 13958 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:47.749598 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.750192 13953 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:47.750895 17209 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:47.749262 17196 consensus_queue.cc:1046] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20250623 01:59:47.751220 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.751485 17209 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.753751 13960 raft_consensus.cc:3053] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:47.754217 13960 raft_consensus.cc:738] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:47.740334 17191 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:47.759593 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:47.764552 13960 consensus_queue.cc:260] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.766165 13960 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:47.768294 16988 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:47.775128 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 93515.8us
I20250623 01:59:47.770280 13954 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 12.
I20250623 01:59:47.779978 17107 raft_consensus.cc:3053] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:47.780450 17107 raft_consensus.cc:738] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:47.780783 17049 raft_consensus.cc:3053] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 LEADER]: Stepping down as leader of term 9
I20250623 01:59:47.781252 17049 raft_consensus.cc:738] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:47.781414 17107 consensus_queue.cc:260] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.782181 17049 consensus_queue.cc:260] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.782799 17107 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:47.783465 17049 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:47.775187 17133 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:47.853358 17206 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:47.854231 17206 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:47.857200 13962 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:47.790625 17191 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:47.843590 16997 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.870230 17133 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:47.890740 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:47.898002 17209 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:47.899302 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 156687us
I20250623 01:59:47.906525 13883 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:47.910396 13960 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:47.910828 17206 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:47.927935 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 216351us
I20250623 01:59:47.908830 17050 raft_consensus.cc:491] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:47.945956 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:47.922533 17106 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
W20250623 01:59:47.947502 17213 log.cc:927] Time spent T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.209s user 0.000s sys 0.000s
W20250623 01:59:47.964552 17214 log.cc:927] Time spent T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.204s user 0.001s sys 0.000s
I20250623 01:59:47.965639 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:47.966924 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:47.967533 17133 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:47.967947 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:47.968506 13886 raft_consensus.cc:2374] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10. Re-sending same reply.
W20250623 01:59:47.946691 17205 log.cc:927] Time spent T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.208s user 0.002s sys 0.000s
I20250623 01:59:47.956710 17206 raft_consensus.cc:491] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:47.957698 17209 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:47.958453 13854 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:47.959292 17211 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.221s user 0.001s sys 0.000s
I20250623 01:59:47.969350 13809 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20250623 01:59:47.971380 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:47.971443 13885 raft_consensus.cc:1238] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250623 01:59:47.972729 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:48.016811 13806 raft_consensus.cc:2408] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250623 01:59:47.942860 17071 consensus_queue.cc:1035] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20250623 01:59:47.965687 13925 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:48.019753 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 54195.6us
I20250623 01:59:47.969291 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:48.011586 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 53204.9us
W20250623 01:59:48.013149 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:48.020406 13886 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:48.021816 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:47.975210 16904 raft_consensus.cc:491] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:48.031859 13955 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:48.025779 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:48.032145 16904 raft_consensus.cc:513] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } }
I20250623 01:59:48.029808 13885 raft_consensus.cc:1238] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:48.032841 13962 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 4.
I20250623 01:59:48.034171 16904 leader_election.cc:290] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:48.015157 17122 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:48.035092 17122 raft_consensus.cc:491] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:48.035490 17122 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
W20250623 01:59:48.035837 17215 log.cc:927] Time spent T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.293s user 0.002s sys 0.000s
I20250623 01:59:48.036131 13854 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.037671 17208 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:48.017362 17209 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:48.038095 17208 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:48.038611 17208 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:48.017355 17206 raft_consensus.cc:513] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.022907 13925 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:48.029795 17020 raft_consensus.cc:3053] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:48.031571 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:48.041716 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:48.038209 13883 raft_consensus.cc:1238] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 8. Current term is 9. Ops: [6.3-8.4]
I20250623 01:59:48.028915 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:48.050530 17206 leader_election.cc:290] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:48.054415 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.057634 13881 raft_consensus.cc:1238] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:48.056222 13882 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20250623 01:59:48.061193 13887 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20250623 01:59:48.068532 17210 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.316s user 0.003s sys 0.000s
I20250623 01:59:48.022218 17049 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:48.041577 17020 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:59:48.056509 17216 log.cc:927] Time spent T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.295s user 0.000s sys 0.002s
I20250623 01:59:48.070806 17020 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:48.072471 13851 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:48.082180 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 59220.7us
I20250623 01:59:48.082487 17068 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:48.083084 17068 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:48.084478 17226 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
W20250623 01:59:48.085644 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 474938us
I20250623 01:59:48.089351 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:48.090421 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:48.091130 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:48.091192 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:48.091584 17133 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:48.092116 17133 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.085065 17068 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:48.094267 17133 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:48.091212 13955 raft_consensus.cc:2408] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20250623 01:59:48.070915 13927 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:48.072046 17020 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:48.114185 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:48.118651 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:48.167006 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:48.162662 13809 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:48.037092 17207 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.062s
I20250623 01:59:47.975792 17106 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.046677 16904 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250623 01:59:48.169571 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 69254us
I20250623 01:59:48.168269 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:48.169520 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 69870.5us
I20250623 01:59:48.015197 17050 raft_consensus.cc:513] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:48.178516 17211 log.cc:927] Time spent T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
W20250623 01:59:48.215406 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 215326us
W20250623 01:59:48.218410 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 50885.4us
I20250623 01:59:48.218824 13776 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5
I20250623 01:59:48.228150 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "632bb79cb33a491fa5765cd80357f1a4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.228997 13925 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:48.229718 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.230057 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.230474 13886 raft_consensus.cc:2391] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 9: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:48.230847 13883 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:48.232054 13853 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:48.223503 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:48.226390 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.235852 13925 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:48.238977 17222 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:48.241775 17106 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:48.242537 17195 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:48.245657 17050 leader_election.cc:290] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:47.965670 17191 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.251456 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:48.252092 13810 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:48.258531 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.210638 17133 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:48.259475 13887 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:48.269029 13925 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:48.227013 17185 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:48.283600 13953 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20250623 01:59:48.225266 17192 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20250623 01:59:48.232024 16904 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20250623 01:59:48.268292 17050 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:48.272498 17013 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:48.277061 17195 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:48.301877 13925 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.265247 17206 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20250623 01:59:48.273202 17133 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.324841 17209 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:48.324786 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:48.300701 17050 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:48.314293 17049 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 01:59:48.347591 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 129061us
I20250623 01:59:48.308243 17013 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:48.333781 17106 consensus_queue.cc:1046] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20250623 01:59:48.296543 17185 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.322628 17192 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:48.352664 17050 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:48.371913 17192 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:48.378798 17192 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:48.380592 13779 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
W20250623 01:59:48.381873 17235 log.cc:927] Time spent T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250623 01:59:48.349615 17195 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.033s
I20250623 01:59:48.383600 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:48.384260 13813 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:48.386054 17105 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:48.384080 17195 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:48.369848 17013 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:48.353969 13853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.4:39393, user_credentials={real_user=slave}} blocked reactor thread for 122027us
W20250623 01:59:48.405069 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 124605us
I20250623 01:59:48.352525 17204 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20250623 01:59:48.412601 17208 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.405893 13810 raft_consensus.cc:2408] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250623 01:59:48.395076 16904 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:48.356596 17049 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.396458 17185 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:48.472414 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 243996us
I20250623 01:59:48.475260 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 01:59:48.445397 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 64894.5us
I20250623 01:59:48.446072 17182 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:48.520460 17182 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:48.523051 17182 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:48.527443 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:48.528153 13805 raft_consensus.cc:2466] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:48.529021 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:48.493888 17233 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20250623 01:59:48.536180 13804 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 01:59:48.531957 13852 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:48.499128 17208 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:48.540026 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 204862us
I20250623 01:59:48.540688 13852 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:48.500471 13779 leader_election.cc:304] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:48.514896 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:48.519485 17133 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:48.543514 13779 leader_election.cc:304] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:48.447755 16904 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.539821 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:48.549304 13779 leader_election.cc:304] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:48.550930 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 332156us
I20250623 01:59:48.550683 17133 raft_consensus.cc:2747] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:48.554359 17105 raft_consensus.cc:2747] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:48.555243 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.555856 13879 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 5.
I20250623 01:59:48.557435 17049 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:48.544278 17243 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250623 01:59:48.558579 17105 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:48.544528 13854 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:48.541338 17245 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:48.571525 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:48.593137 13776 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.616819 13776 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.547423 16904 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:48.626109 17114 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:48.626679 17114 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:48.627111 17114 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:48.557147 17050 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:48.648624 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 77300.4us
W20250623 01:59:48.654253 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 62170.6us
I20250623 01:59:48.593806 16997 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250623 01:59:48.617007 17236 log.cc:927] Time spent T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250623 01:59:48.613233 17208 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:48.642186 17133 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election won for term 8
W20250623 01:59:48.672056 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 57734.6us
W20250623 01:59:48.648965 17233 log.cc:927] Time spent T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20250623 01:59:48.641386 17001 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:48.674979 13776 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.675029 13852 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:48.684226 14478 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.694203 17095 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:48.694620 17050 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:48.696579 17107 raft_consensus.cc:3053] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:48.696992 17107 raft_consensus.cc:738] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:48.697816 17107 consensus_queue.cc:260] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.699169 17107 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:48.707284 17248 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:48.707708 17248 raft_consensus.cc:491] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:48.708026 17248 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:48.709288 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.682443 17001 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.711452 17114 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.711812 17001 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:48.712956 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:48.713579 13958 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
W20250623 01:59:48.713753 17020 consensus_meta.cc:287] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.572s user 0.005s sys 0.006s
I20250623 01:59:48.686659 17192 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.716549 13854 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.716933 17192 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:48.716571 13852 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:48.721954 13880 raft_consensus.cc:1273] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250623 01:59:48.722162 16870 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:48.725415 17114 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:48.725790 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:48.743112 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.744411 13852 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:48.694694 17095 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:48.728129 17013 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:48.733657 17182 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:48.748414 17095 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:48.752974 17182 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:48.753463 17182 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:48.754076 17248 raft_consensus.cc:513] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:48.756878 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:48.757500 13880 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20250623 01:59:48.758069 17122 consensus_meta.cc:287] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Time spent flushing consensus metadata: real 0.721s user 0.007s sys 0.003s
I20250623 01:59:48.758540 17122 raft_consensus.cc:513] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:48.750938 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:48.764117 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:48.765374 17122 leader_election.cc:290] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:48.758555 17248 leader_election.cc:290] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:48.741044 16870 raft_consensus.cc:491] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:48.770334 16870 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:48.749899 16988 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 01:59:48.750157 17105 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20250623 01:59:48.770090 17185 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:48.746727 17243 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250623 01:59:48.768069 17122 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20250623 01:59:48.740218 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:48.783195 13887 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:48.786255 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:48.787418 13805 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:48.787736 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:48.789367 13813 raft_consensus.cc:2366] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6.
I20250623 01:59:48.788376 13811 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
W20250623 01:59:48.791707 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 309580us
I20250623 01:59:48.796015 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c708cd61006343f09750b88b2621eec6" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:48.811690 13807 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:48.832594 14479 leader_election.cc:400] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781) with higher term. Message: Invalid argument: T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6.
W20250623 01:59:48.837831 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 288602us
I20250623 01:59:48.844597 13779 leader_election.cc:304] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:48.881281 13779 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:48.884526 13887 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 11.
I20250623 01:59:48.885776 13776 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.912717 13779 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:48.851730 17242 raft_consensus.cc:2747] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:48.921041 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.921725 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.922848 13887 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 10. Current term is 11. Ops: [8.7-10.8]
I20250623 01:59:48.923255 13883 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250623 01:59:48.923755 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.924383 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:48.925019 13883 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:48.853914 17068 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20250623 01:59:48.926623 13885 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250623 01:59:48.925091 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.927816 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.931618 13807 raft_consensus.cc:2466] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 11.
I20250623 01:59:48.947268 13887 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:48.846779 17133 raft_consensus.cc:695] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:48.948204 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 01:59:48.949661 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:48.974104 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:48.973584 17242 consensus_queue.cc:1046] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250623 01:59:48.981663 13881 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
W20250623 01:59:48.971484 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:48.987001 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:48.918692 17263 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 01:59:48.926716 17105 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250623 01:59:48.988826 17258 log.cc:927] Time spent T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.109s user 0.001s sys 0.001s
I20250623 01:59:48.952615 17185 raft_consensus.cc:2747] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250623 01:59:49.000450 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 95205.5us
I20250623 01:59:48.986574 17206 raft_consensus.cc:2747] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250623 01:59:49.006289 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 85871.1us
I20250623 01:59:49.029597 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:49.020746 17013 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:49.034554 17259 log.cc:927] Time spent T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250623 01:59:49.043565 16870 raft_consensus.cc:513] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.051712 13805 raft_consensus.cc:2466] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:49.055954 13779 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:49.062129 13779 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:49.063186 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:49.064692 13881 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:49.075045 13883 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:49.069190 13776 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:48.998618 17257 consensus_queue.cc:1046] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
W20250623 01:59:49.079255 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 64056.7us
I20250623 01:59:49.078763 13852 leader_election.cc:304] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:49.069418 17105 raft_consensus.cc:491] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:49.103847 16886 raft_consensus.cc:2802] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 01:59:49.104496 16886 raft_consensus.cc:695] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:49.108093 16886 consensus_queue.cc:237] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.077960 17050 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:49.113253 13852 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters: c243715fe12445019c33bbeafbaea635
I20250623 01:59:49.115916 17050 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.116323 17122 raft_consensus.cc:2802] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:49.117060 17122 raft_consensus.cc:695] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:49.118106 17122 consensus_queue.cc:237] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.096549 17133 consensus_queue.cc:237] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.093595 17068 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20250623 01:59:49.076999 17253 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250623 01:59:49.081791 16870 leader_election.cc:290] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.131484 17114 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20250623 01:59:49.132170 17114 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:49.132622 17114 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Advancing to term 13
I20250623 01:59:49.127666 17222 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:49.134333 17222 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.136979 17222 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.139359 17105 raft_consensus.cc:513] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.141844 17105 leader_election.cc:290] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.128047 17257 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:49.143438 17257 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.161082 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:49.161890 13962 raft_consensus.cc:2391] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 7: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:49.164644 13925 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:49.170359 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:49.178129 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:49.180603 13811 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 7
I20250623 01:59:49.189548 13883 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 10. Current term is 11. Ops: []
I20250623 01:59:49.130167 17279 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:49.198642 17263 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:49.157522 17253 raft_consensus.cc:491] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:49.178472 17106 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:49.176560 17105 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:49.147687 17068 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20250623 01:59:49.208580 17106 raft_consensus.cc:491] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:49.210171 17253 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:59:49.214330 17271 log.cc:927] Time spent T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.133s user 0.003s sys 0.000s
I20250623 01:59:49.212740 17106 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:49.147373 13959 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:49.147179 17095 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.221923 17050 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:49.183888 17251 consensus_queue.cc:1046] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20250623 01:59:49.222980 17257 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:49.179744 17192 consensus_queue.cc:1046] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20250623 01:59:49.178045 17222 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:49.234436 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 65684.6us
I20250623 01:59:49.147013 17182 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:49.297454 17182 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:49.233104 17114 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.302300 13804 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 12. Current term is 13. Ops: []
I20250623 01:59:49.303906 17114 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 13 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.312263 14515 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:49.237672 13811 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:49.223783 17105 raft_consensus.cc:695] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:49.195897 17279 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:49.273527 13925 leader_election.cc:304] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:49.275355 17095 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.288681 17191 raft_consensus.cc:3053] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:49.329217 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:49.336928 13884 raft_consensus.cc:2366] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6.
I20250623 01:59:49.325192 13852 leader_election.cc:304] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:49.331084 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:49.273057 17107 consensus_meta.cc:287] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635: Time spent flushing consensus metadata: real 0.554s user 0.013s sys 0.000s
W20250623 01:59:49.340302 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 66836.3us
W20250623 01:59:49.364830 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.356309 17248 raft_consensus.cc:2802] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:49.381712 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:49.368947 13959 raft_consensus.cc:604] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:49.408311 13959 raft_consensus.cc:612] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:49.332798 17257 raft_consensus.cc:3053] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:49.327677 17225 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:49.330312 17279 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:49.336163 17050 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:49.325343 17222 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.336083 17191 raft_consensus.cc:738] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:49.420779 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 353969us
I20250623 01:59:49.398802 17251 raft_consensus.cc:2802] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:49.421478 13779 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:49.350528 14478 leader_election.cc:400] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6.
I20250623 01:59:49.407052 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.474530 14478 leader_election.cc:403] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250623 01:59:49.379482 17122 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:49.411753 17257 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:49.454666 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:49.482061 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 58113.2us
I20250623 01:59:49.412297 13879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:49.475800 13852 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:49.434053 14515 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
W20250623 01:59:49.444795 17278 log.cc:927] Time spent T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
W20250623 01:59:49.434995 17261 log.cc:927] Time spent T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.382s user 0.000s sys 0.005s
W20250623 01:59:49.489005 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 147626us
I20250623 01:59:49.493772 17243 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:49.435626 17133 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:49.438972 17253 raft_consensus.cc:513] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.442013 17222 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.443486 17192 raft_consensus.cc:3053] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:49.504446 13779 leader_election.cc:304] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:49.199916 17263 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.447993 17251 raft_consensus.cc:695] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [term 6 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
W20250623 01:59:49.529683 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 141217us
I20250623 01:59:49.420361 17248 raft_consensus.cc:695] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:49.513653 17192 raft_consensus.cc:738] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:49.500645 17133 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:49.522888 17253 leader_election.cc:290] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.422200 17225 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:49.524650 17269 log.cc:927] Time spent T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
I20250623 01:59:49.525353 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:49.574046 13960 raft_consensus.cc:3058] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:49.526180 16886 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:49.525791 17257 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.528496 13779 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:49.447470 17286 consensus_queue.cc:1046] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.098s
I20250623 01:59:49.534904 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:49.537217 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.532703 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:49.424798 17050 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.556931 17252 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:49.573196 13885 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 10. Current term is 11. Ops: [8.7-10.8]
W20250623 01:59:49.533485 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.501083 17051 raft_consensus.cc:2747] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d22741cfb9b408684cb82c58241b2f5 for earlier term 5. Current term is 6.
I20250623 01:59:49.567170 17013 raft_consensus.cc:2747] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250623 01:59:49.583262 17261 log.cc:927] Time spent T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250623 01:59:49.552974 17133 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:49.484822 17122 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.593433 13882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58136 (request call id 2346) took 1367 ms. Trace:
W20250623 01:59:49.594265 17283 log.cc:927] Time spent T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
W20250623 01:59:49.594811 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 66401.5us
I20250623 01:59:49.594938 17253 raft_consensus.cc:2747] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:49.607775 13882 rpcz_store.cc:276] 0623 01:59:48.225595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:48.225718 (+ 123us) service_pool.cc:225] Handling call
0623 01:59:48.261023 (+ 35305us) raft_consensus.cc:1469] Updating replica for 1 ops
0623 01:59:48.396762 (+135739us) raft_consensus.cc:1532] Early marking committed up to index 4
0623 01:59:48.442028 (+ 45266us) log.cc:844] Serialized 14 byte log entry
0623 01:59:48.719394 (+277366us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0623 01:59:48.719861 (+ 467us) log.cc:844] Serialized 26 byte log entry
0623 01:59:48.719998 (+ 137us) raft_consensus.cc:1643] Marking committed up to 4
0623 01:59:48.720030 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0623 01:59:48.720089 (+ 59us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0623 01:59:49.434481 (+714392us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:49.593102 (+158621us) raft_consensus.cc:1683] finished
0623 01:59:49.593192 (+ 90us) raft_consensus.cc:1689] UpdateReplicas() finished
0623 01:59:49.593351 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46032,"prepare.queue_time_us":49216,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":48,"raft.queue_time_us":65274,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":33071,"spinlock_wait_cycles":24303488,"thread_start_us":314978,"threads_started":2,"wal-append.queue_time_us":478668}
I20250623 01:59:49.621697 13884 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250623 01:59:49.595882 17251 consensus_queue.cc:237] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.612195 13779 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:49.429822 17191 consensus_queue.cc:260] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.597702 17192 consensus_queue.cc:260] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:49.599085 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 224679us
I20250623 01:59:49.600165 17286 consensus_queue.cc:1230] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer ee041a8325864e788a383b8356afc652 log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250623 01:59:49.601701 17266 raft_consensus.cc:3053] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:49.589056 17252 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.643919 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:49.644606 17185 raft_consensus.cc:2747] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:49.645056 17252 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:49.645619 17252 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:49.646062 17252 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.646749 17276 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:49.606216 17279 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.649135 13812 raft_consensus.cc:1238] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:49.435868 17105 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:49.653059 17122 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.655092 17225 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.655406 13809 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:49.656145 17192 raft_consensus.cc:3058] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:49.509092 17277 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
I20250623 01:59:49.649061 17279 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.635478 13807 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20250623 01:59:49.651492 17270 log.cc:927] Time spent T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.575s user 0.000s sys 0.003s
I20250623 01:59:49.639741 17248 consensus_queue.cc:237] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.662575 17252 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:49.621762 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 214271us
I20250623 01:59:49.662786 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:49.663681 13883 raft_consensus.cc:2391] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 5: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:49.665940 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:49.650941 13805 raft_consensus.cc:1238] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 9. Current term is 10. Ops: []
I20250623 01:59:49.669757 13805 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250623 01:59:49.672852 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.675552 17252 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:49.676165 17252 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:49.682952 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.685092 13812 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:49.641273 17266 raft_consensus.cc:738] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:49.714308 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 121) took 7519 ms. Trace:
I20250623 01:59:49.645047 17106 raft_consensus.cc:513] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.684315 17191 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:49.737804 17133 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.717523 13806 raft_consensus.cc:1238] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:49.681965 17042 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.037s
I20250623 01:59:49.719470 17106 leader_election.cc:290] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:49.720446 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:49.720440 17122 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250623 01:59:49.725790 17257 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:49.726300 17266 consensus_queue.cc:260] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:49.729331 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 252551us
I20250623 01:59:49.730316 17050 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:49.736514 17252 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:49.680076 17209 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:49.742399 13810 raft_consensus.cc:1238] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 5. Current term is 6. Ops: []
I20250623 01:59:49.748165 13807 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:49.748338 13852 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:49.759372 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.779402 13960 raft_consensus.cc:2466] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
W20250623 01:59:49.789772 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 94801.4us
W20250623 01:59:49.802995 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.716418 13708 rpcz_store.cc:276] 0623 01:59:42.195320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:42.195479 (+ 159us) service_pool.cc:225] Handling call
0623 01:59:49.714206 (+7518727us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:47.469462 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:47.470145 (+ 683us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:47.470197 (+ 52us) write_op.cc:435] Acquired schema lock
0623 01:59:47.470215 (+ 18us) tablet.cc:660] Decoding operations
0623 01:59:47.471082 (+ 867us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:47.471170 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:47.471210 (+ 40us) tablet.cc:683] Acquiring locks for 6 operations
0623 01:59:47.471806 (+ 596us) tablet.cc:699] Row locks acquired
0623 01:59:47.471823 (+ 17us) write_op.cc:260] PREPARE: finished
0623 01:59:47.471996 (+ 173us) write_op.cc:270] Start()
0623 01:59:47.472129 (+ 133us) write_op.cc:276] Timestamp: P: 1750643987471931 usec, L: 0
0623 01:59:47.472163 (+ 34us) op_driver.cc:348] REPLICATION: starting
0623 01:59:47.472908 (+ 745us) log.cc:844] Serialized 2477 byte log entry
0623 01:59:48.067194 (+594286us) spinlock_profiling.cc:243] Waited 95 ms on lock 0x7b440002494c. stack: 00007f41e0b0e493 00007f41e0b0e390 00007f41e0467574 00007f41e9f5b5a4 00007f41e9f5b539 00007f41e0983971 00007f41e0a51fc5 00007f41e0a529e2 00007f41e36eefc1 00007f41e36ea8ea 00007f41e36de6ae 00007f41e36dd9f6 00007f41e36dd618 00007f41e36e7051 00007f41e36e7009 00007f41e36e6f99
0623 01:59:48.476943 (+409749us) op_driver.cc:464] REPLICATION: finished
0623 01:59:48.918412 (+441469us) write_op.cc:301] APPLY: starting
0623 01:59:48.949216 (+ 30804us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:49.002055 (+ 52839us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:49.018966 (+ 16911us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:49.110801 (+ 91835us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:49.111113 (+ 312us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0623 01:59:49.111173 (+ 60us) write_op.cc:312] APPLY: finished
0623 01:59:49.111464 (+ 291us) log.cc:844] Serialized 53 byte log entry
0623 01:59:49.111630 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:49.111738 (+ 108us) write_op.cc:454] Released schema lock
0623 01:59:49.111803 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:49.112235 (+ 432us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:49.112263 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":145045,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":344023,"apply.run_cpu_time_us":3408,"apply.run_wall_time_us":206894,"num_ops":6,"prepare.queue_time_us":436873,"prepare.run_cpu_time_us":24531,"prepare.run_wall_time_us":210269,"raft.queue_time_us":313085,"raft.run_cpu_time_us":27436,"raft.run_wall_time_us":289377,"replication_time_us":941620,"spinlock_wait_cycles":218588032,"thread_start_us":842430,"threads_started":4,"wal-append.queue_time_us":257263}]]}
I20250623 01:59:49.819789 13811 raft_consensus.cc:1238] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 8. Current term is 9. Ops: []
W20250623 01:59:49.825739 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 50643.9us
I20250623 01:59:49.802534 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.852416 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:49.853269 13882 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:49.854732 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:49.845672 13806 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 5.
I20250623 01:59:49.860298 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.799136 16870 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 01:59:49.863797 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:49.863899 13812 raft_consensus.cc:1238] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 10. Current term is 11. Ops: []
I20250623 01:59:49.864415 13804 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:49.905541 13810 raft_consensus.cc:1238] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 12. Current term is 13. Ops: []
I20250623 01:59:49.907568 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
W20250623 01:59:49.864388 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.863233 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.868166 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.872216 17243 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:49.878006 17285 raft_consensus.cc:3053] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:49.883875 17050 raft_consensus.cc:491] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:49.858245 17252 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:49.844326 17266 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:49.920912 13807 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:49.818286 17106 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
W20250623 01:59:49.923854 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.924750 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:49.933995 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.932094 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:49.934718 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:49.927968 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.934641 13955 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:49.938060 13883 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:49.939587 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:49.904084 16870 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:49.925935 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:49.944756 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:49.941169 13955 raft_consensus.cc:2391] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 10: Already voted for candidate c243715fe12445019c33bbeafbaea635 in this term.
I20250623 01:59:49.944221 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.946008 13811 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:49.844046 17209 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.914289 17285 raft_consensus.cc:738] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:49.952172 17285 consensus_queue.cc:260] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.941267 17133 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.941468 17050 raft_consensus.cc:513] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.952677 13882 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:49.949272 13885 raft_consensus.cc:2366] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:49.925701 17297 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:49.938236 17252 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.915356 16988 consensus_queue.cc:1046] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.038s
I20250623 01:59:49.967828 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:49.947212 16870 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
W20250623 01:59:49.969640 13778 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:49.970118 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:49.957221 17302 consensus_queue.cc:1046] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.016s
I20250623 01:59:49.971073 13710 catalog_manager.cc:5582] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 7 to 8, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 8 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:49.972954 13710 catalog_manager.cc:5582] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 7, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:49.974637 13710 catalog_manager.cc:5582] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 5, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:49.977970 13880 raft_consensus.cc:2391] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 6: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:49.978848 17209 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.980816 13806 raft_consensus.cc:1238] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Rejecting Update request from peer c243715fe12445019c33bbeafbaea635 for earlier term 10. Current term is 11. Ops: []
I20250623 01:59:49.981444 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.981778 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.982365 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.982976 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:49.983172 17252 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:49.984465 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:49.985033 13811 raft_consensus.cc:3058] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:49.981837 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.983564 13806 raft_consensus.cc:3058] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:49.987627 16870 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:49.990940 16870 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:49.992920 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:49.993192 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.993439 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.992277 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:49.997355 13806 raft_consensus.cc:2466] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:49.993534 13804 raft_consensus.cc:2374] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9. Re-sending same reply.
I20250623 01:59:50.000169 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.007354 17050 leader_election.cc:290] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:49.977208 13710 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 6, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:50.011113 13710 catalog_manager.cc:5582] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 6 to 7, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:50.012037 17295 raft_consensus.cc:3053] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 LEADER]: Stepping down as leader of term 12
I20250623 01:59:50.029606 17295 raft_consensus.cc:738] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:50.030491 17295 consensus_queue.cc:260] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.031702 17295 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Advancing to term 13
I20250623 01:59:49.945890 17106 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:50.038196 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.039290 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.019637 13925 leader_election.cc:304] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:50.040057 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e773c5700b0945d1981f481534952e85" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.040289 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.040794 13959 raft_consensus.cc:2366] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7.
I20250623 01:59:50.041491 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.042160 17119 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:50.042418 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44146f651124d8fba4cc1825f480603" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.042708 17119 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:50.042322 13958 raft_consensus.cc:2408] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20250623 01:59:50.045967 17119 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:50.046653 13811 raft_consensus.cc:2466] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:50.046849 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.047061 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.047753 13958 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 6.
I20250623 01:59:50.048420 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:50.049331 17106 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:49.969385 17285 raft_consensus.cc:3058] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:50.024201 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.021755 13813 raft_consensus.cc:2408] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250623 01:59:50.051457 13806 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:50.028234 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:50.054327 13804 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:50.057977 13953 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20250623 01:59:50.058693 13961 raft_consensus.cc:1273] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250623 01:59:50.059063 13961 pending_rounds.cc:85] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 3
I20250623 01:59:50.059265 13961 pending_rounds.cc:107] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250623 01:59:50.059301 13956 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20250623 01:59:50.059582 13961 raft_consensus.cc:2887] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:50.020813 17245 raft_consensus.cc:3053] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:50.065255 14478 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.065495 17245 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:50.066049 14511 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250623 01:59:50.066224 17119 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:50.066399 17245 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.030491 13710 catalog_manager.cc:5582] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 6 to 7, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:50.052774 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.059690 13956 pending_rounds.cc:85] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 5
I20250623 01:59:50.069155 13956 pending_rounds.cc:107] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20250623 01:59:50.069438 13956 raft_consensus.cc:2887] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:50.070757 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.071053 17266 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:50.071645 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:50.072923 13804 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
I20250623 01:59:49.966671 17297 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.074155 13954 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:50.075428 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86045b40aeea4a02993bcd65deecd670" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:50.074110 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.076905 13961 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:50.076926 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:50.067718 17245 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:50.060398 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.081410 13710 catalog_manager.cc:5582] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 5, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:50.045212 17257 raft_consensus.cc:2802] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:50.089258 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.086491 17297 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:50.089015 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05203153a0ed43df91fde4fe090579a7" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.094786 13953 raft_consensus.cc:2408] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20250623 01:59:50.090171 13954 raft_consensus.cc:1238] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250623 01:59:50.090425 13882 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20250623 01:59:50.043332 13959 raft_consensus.cc:2408] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250623 01:59:50.084431 13776 leader_election.cc:400] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
I20250623 01:59:50.090993 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.096028 13962 raft_consensus.cc:1273] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20250623 01:59:50.096545 13962 pending_rounds.cc:85] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 3
I20250623 01:59:50.098376 13962 pending_rounds.cc:107] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20250623 01:59:50.108199 13962 raft_consensus.cc:2887] T f44146f651124d8fba4cc1825f480603 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:50.127640 13779 leader_election.cc:400] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7.
I20250623 01:59:50.128016 13779 leader_election.cc:403] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:50.128643 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.135375 13776 leader_election.cc:403] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250623 01:59:50.139248 13953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.139990 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.140543 13958 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
W20250623 01:59:50.141543 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:50.142432 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 45319.5us
I20250623 01:59:50.143673 13807 raft_consensus.cc:1238] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 6. Current term is 7. Ops: []
I20250623 01:59:50.144845 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:50.151278 13710 catalog_manager.cc:5582] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 7 to 8, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:50.152382 13927 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.156621 13959 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20250623 01:59:50.157006 13959 pending_rounds.cc:85] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 4
I20250623 01:59:50.157198 13959 pending_rounds.cc:107] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20250623 01:59:50.157490 13959 raft_consensus.cc:2887] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:50.157292 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:50.160804 14515 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:50.155961 17122 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:50.161711 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.175721 13958 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
W20250623 01:59:50.178579 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.182767 17304 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250623 01:59:50.169900 17257 raft_consensus.cc:695] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:50.186942 13956 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2f02354fc186482aa0fc28cab32390c5"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:50.187500 13956 raft_consensus.cc:604] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:50.187835 13956 raft_consensus.cc:612] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:50.189910 13852 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:50.190943 13852 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
W20250623 01:59:50.193128 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.193487 17207 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:50.193982 17207 raft_consensus.cc:491] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:50.194245 16870 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:50.194384 17207 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:50.194800 16870 raft_consensus.cc:695] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:50.195617 16870 consensus_queue.cc:237] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.155526 17185 raft_consensus.cc:2747] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 6. Current term is 7.
I20250623 01:59:50.200536 13960 raft_consensus.cc:1273] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20250623 01:59:50.203058 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 55402.3us
I20250623 01:59:50.206465 13710 catalog_manager.cc:5582] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:50.207412 13779 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
W20250623 01:59:50.207146 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 122850us
I20250623 01:59:50.213894 13956 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:50.214519 13956 raft_consensus.cc:604] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:50.214803 13956 raft_consensus.cc:612] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:50.214926 13779 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.212836 17185 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20250623 01:59:50.220117 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:50.216583 17263 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250623 01:59:50.241591 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.212980 17105 raft_consensus.cc:2747] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ee041a8325864e788a383b8356afc652 (127.12.87.2:43055) with higher term. Message: Invalid argument: T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 5. Current term is 6.
W20250623 01:59:50.267561 13776 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.244438 13880 raft_consensus.cc:1273] T 5d4cc11489de46dcb7d5f77daefd49af P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:50.205933 17133 consensus_queue.cc:1046] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250623 01:59:50.228358 17013 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:50.213152 17279 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20250623 01:59:50.184118 17257 consensus_queue.cc:237] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.291193 13956 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:42192
I20250623 01:59:50.291988 13956 raft_consensus.cc:604] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:50.292403 13956 raft_consensus.cc:612] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:50.296305 13854 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:50.307452 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250623 01:59:50.278115 17277 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.031s
I20250623 01:59:50.316259 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:50.311355 17021 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:50.311177 17286 consensus_queue.cc:1035] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20250623 01:59:50.344954 17021 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:50.325030 17248 consensus_queue.cc:1035] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20250623 01:59:50.366042 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:50.366583 17021 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:50.407264 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 49242.3us
I20250623 01:59:50.417430 13959 raft_consensus.cc:1273] T e773c5700b0945d1981f481534952e85 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:50.365166 17207 raft_consensus.cc:513] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.418988 17204 consensus_queue.cc:1035] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.042s
I20250623 01:59:50.412918 17304 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:50.416512 17302 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:50.469558 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.481702 13960 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:50.490036 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:50.490677 13812 raft_consensus.cc:3058] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:50.469914 17304 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.500906 17304 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:50.460999 17204 consensus_queue.cc:1230] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 [LEADER]: Peer c243715fe12445019c33bbeafbaea635 log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250623 01:59:50.514153 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa785c3afe64dbcac3b658b49195b8c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.525126 13956 pending_rounds.cc:85] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 7
I20250623 01:59:50.525633 13956 pending_rounds.cc:107] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20250623 01:59:50.526005 13956 raft_consensus.cc:2887] T c708cd61006343f09750b88b2621eec6 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:50.535897 13812 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
W20250623 01:59:50.538100 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.547258 13959 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.460163 17207 leader_election.cc:290] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:50.457242 17122 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250623 01:59:50.465373 17296 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.132s
I20250623 01:59:50.565637 13852 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:50.487640 17302 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.570283 13883 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10. Ops: []
I20250623 01:59:50.533911 17021 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.569911 13960 raft_consensus.cc:2466] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:50.586678 17250 raft_consensus.cc:2802] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:50.588701 17302 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:50.590617 17021 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:50.595522 17250 raft_consensus.cc:695] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:50.599537 13954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:50.604229 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:50.606745 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.608616 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
W20250623 01:59:50.609375 13925 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.610270 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.608424 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:50.611814 13885 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:50.596591 17250 consensus_queue.cc:237] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.616907 13812 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250623 01:59:50.637491 13852 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:50.660715 13852 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:50.682448 13925 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:50.658360 17248 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 01:59:50.686198 13812 raft_consensus.cc:1273] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:50.692570 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.698468 13804 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
W20250623 01:59:50.698805 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 171048us
I20250623 01:59:50.699575 13883 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250623 01:59:50.694993 13779 leader_election.cc:304] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.693353 13806 raft_consensus.cc:1273] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250623 01:59:50.709057 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:50.724126 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.681128 17303 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250623 01:59:50.735970 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250623 01:59:50.737709 13926 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.732697 17257 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250623 01:59:50.741573 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 48813.6us
I20250623 01:59:50.743561 13779 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.746793 13779 leader_election.cc:304] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.750777 13880 raft_consensus.cc:1238] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:50.741220 17276 raft_consensus.cc:3053] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 LEADER]: Stepping down as leader of term 9
I20250623 01:59:50.759963 17276 raft_consensus.cc:738] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:50.741495 17257 raft_consensus.cc:491] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:50.694186 17243 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20250623 01:59:50.692914 17320 consensus_queue.cc:1046] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20250623 01:59:50.758067 17105 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250623 01:59:50.755246 17263 raft_consensus.cc:2747] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:50.765537 17257 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
W20250623 01:59:50.773910 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 44515.8us
I20250623 01:59:50.782790 17224 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:50.783356 17224 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.785928 13885 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:50.768312 17021 consensus_queue.cc:1035] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250623 01:59:50.790343 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:50.791016 13955 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:50.796135 17263 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:50.800158 13854 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:50.766237 17285 consensus_queue.cc:1046] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20250623 01:59:50.799073 17276 consensus_queue.cc:260] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:50.806114 13880 raft_consensus.cc:1238] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:50.801277 13779 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.813755 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:50.807289 13957 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250623 01:59:50.810894 17263 consensus_queue.cc:1046] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
W20250623 01:59:50.849563 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 91409.1us
W20250623 01:59:50.868382 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 181402us
I20250623 01:59:50.808881 16870 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:50.872984 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.838517 17105 raft_consensus.cc:2747] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:50.811554 17276 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:50.801421 17224 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:50.849285 17286 raft_consensus.cc:3053] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 9 LEADER]: Stepping down as leader of term 9
I20250623 01:59:50.839402 17323 raft_consensus.cc:2747] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20250623 01:59:50.882012 13926 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:50.816648 17013 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20250623 01:59:50.854027 16886 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
W20250623 01:59:50.904973 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 60460.5us
I20250623 01:59:50.875365 16870 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:50.902575 17266 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20250623 01:59:50.896592 17319 consensus_queue.cc:1046] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20250623 01:59:50.888521 17286 raft_consensus.cc:738] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:50.916731 16870 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:50.934111 17286 consensus_queue.cc:260] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:50.945087 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 58213.5us
I20250623 01:59:50.968402 17285 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:50.960986 17286 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
W20250623 01:59:50.976493 13928 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:50.968832 17257 raft_consensus.cc:513] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:51.054847 17346 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20250623 01:59:50.973991 17304 raft_consensus.cc:2747] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250623 01:59:51.131034 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 50548.5us
W20250623 01:59:51.137588 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 55149.9us
W20250623 01:59:51.150282 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 262375us
W20250623 01:59:51.155434 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 273577us
I20250623 01:59:51.156551 13926 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.158598 13926 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.160030 13926 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.164887 13926 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.160272 17304 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20250623 01:59:51.283406 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 118580us
W20250623 01:59:51.251888 14479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 93471.7us
I20250623 01:59:51.288216 13926 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.163473 17353 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:51.298126 13883 raft_consensus.cc:1273] T 839cb616e68044308dfa7e08eadc1903 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:51.327744 13926 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.286543 17320 raft_consensus.cc:491] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:51.329034 13926 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.284981 17347 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 4 FOLLOWER]: Advancing to term 5
W20250623 01:59:51.332638 13777 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 01:59:51.333169 17304 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:51.335666 17304 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250623 01:59:51.337123 17357 log.cc:927] Time spent T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250623 01:59:51.330848 17303 raft_consensus.cc:491] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:51.343338 17303 raft_consensus.cc:513] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:51.343637 17320 raft_consensus.cc:513] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:51.346199 17320 leader_election.cc:290] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:51.333505 17349 log.cc:927] Time spent T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.371s user 0.001s sys 0.000s
I20250623 01:59:51.346897 17303 leader_election.cc:290] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:51.362025 17277 raft_consensus.cc:3053] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:51.362509 17277 raft_consensus.cc:738] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:51.363381 17277 consensus_queue.cc:260] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.364516 17277 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:51.331194 17348 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:51.379828 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:51.394459 17276 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:51.395103 17276 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.399091 17276 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:51.410367 17133 raft_consensus.cc:3053] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:51.410830 17133 raft_consensus.cc:738] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:51.411660 17133 consensus_queue.cc:260] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.412732 17133 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:51.327944 17358 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:51.357649 17257 leader_election.cc:290] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:51.444455 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 66511.9us
I20250623 01:59:51.458495 13879 raft_consensus.cc:1238] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:51.461374 13881 raft_consensus.cc:1238] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:51.462569 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:51.465176 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 87043.3us
I20250623 01:59:51.467288 17050 consensus_queue.cc:1035] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:51.469702 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:51.470034 13807 raft_consensus.cc:1273] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:51.470359 13886 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:51.472278 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:51.473965 13776 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:51.357534 16870 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.475499 17133 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:51.475898 17133 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:51.476117 13882 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20250623 01:59:51.476269 17133 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:51.479175 16870 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:51.488440 13955 raft_consensus.cc:1273] T 1aa785c3afe64dbcac3b658b49195b8c P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250623 01:59:51.490623 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 161661us
I20250623 01:59:51.493654 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:51.494216 13882 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:51.497037 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:51.497648 13958 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:51.502069 17323 consensus_queue.cc:1046] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:51.503288 17266 raft_consensus.cc:3053] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:51.503639 17266 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:51.504421 17266 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.506016 17266 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:51.507779 17250 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:51.491955 13926 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters: ee041a8325864e788a383b8356afc652
I20250623 01:59:51.513056 13882 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:51.514118 13958 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:51.517068 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:51.521106 13854 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:51.523921 17362 consensus_queue.cc:1035] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20250623 01:59:51.524688 17133 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.525473 13926 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:51.519073 16988 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.005s sys 0.013s
I20250623 01:59:51.528281 13852 leader_election.cc:304] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:51.538648 13926 leader_election.cc:304] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 01:59:51.529721 16988 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.005s sys 0.013s
I20250623 01:59:51.561757 13926 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.563300 13926 leader_election.cc:304] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.537551 17250 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:51.537981 17248 raft_consensus.cc:2747] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:51.569460 13926 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.540843 17050 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250623 01:59:51.614464 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 45119.6us
I20250623 01:59:51.617175 13880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:51.617919 13880 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:51.623322 14515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:51.568756 17358 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:51.570415 17348 raft_consensus.cc:2747] T 2f02354fc186482aa0fc28cab32390c5 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250623 01:59:51.637643 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 68514.5us
I20250623 01:59:51.613801 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2928af9e5df4f9eacf59f499ab96dc1" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:51.620740 17341 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 01:59:51.619233 17285 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:51.622510 17257 raft_consensus.cc:2762] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20250623 01:59:51.647706 17365 raft_consensus.cc:2747] T 86045b40aeea4a02993bcd65deecd670 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250623 01:59:51.648876 17368 log.cc:927] Time spent T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250623 01:59:51.634577 17050 consensus_queue.cc:1035] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20250623 01:59:51.655064 13778 leader_election.cc:304] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.655848 17320 raft_consensus.cc:2747] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 01:59:51.660295 17285 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.662444 17285 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:51.665537 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 49474us
I20250623 01:59:51.669996 13925 leader_election.cc:304] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:51.670939 17257 raft_consensus.cc:2802] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 01:59:51.671412 17257 raft_consensus.cc:695] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:51.672214 17257 consensus_queue.cc:237] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.673439 17133 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:51.677582 13880 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
I20250623 01:59:51.680596 13776 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:51.687906 13813 raft_consensus.cc:2408] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250623 01:59:51.695158 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:51.697405 13884 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 13.
I20250623 01:59:51.698217 13925 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:51.709951 17347 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250623 01:59:51.692539 17122 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:51.687367 17250 raft_consensus.cc:695] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:51.692786 17277 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election won for term 8
W20250623 01:59:51.722648 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 99573.9us
I20250623 01:59:51.723572 13779 leader_election.cc:304] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.705646 17365 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20250623 01:59:51.730568 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e1fc4335c84e18ba2e5c6ea55fed7a" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:51.731674 13960 raft_consensus.cc:1238] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20250623 01:59:51.732508 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:51.733256 13956 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 7.
I20250623 01:59:51.735167 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "839cb616e68044308dfa7e08eadc1903" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:51.735446 13779 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.735678 13956 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer 85a58a834b6b4b0dbe197b3dad87e382 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20250623 01:59:51.736577 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d4cc11489de46dcb7d5f77daefd49af" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:51.735474 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:51.738234 13806 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 13: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:51.735193 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e07b6deb3546fcad55d41d2ecd8a5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:51.738866 13804 raft_consensus.cc:3058] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:51.739893 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:51.740584 13955 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:51.740640 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:51.737279 13810 raft_consensus.cc:2374] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6. Re-sending same reply.
I20250623 01:59:51.735828 13805 raft_consensus.cc:2374] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8. Re-sending same reply.
I20250623 01:59:51.746847 13809 raft_consensus.cc:1273] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 01:59:51.748503 17250 consensus_queue.cc:237] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.731952 17277 raft_consensus.cc:695] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:51.750280 17277 consensus_queue.cc:237] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.752749 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d3844327a1b452da89d70d856d0a80f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:51.731030 17358 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:51.756700 17358 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.758694 17358 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:51.764119 13804 raft_consensus.cc:2466] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:51.771499 13955 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 8.
W20250623 01:59:51.773437 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 45765.3us
I20250623 01:59:51.786868 13961 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20250623 01:59:51.731676 17365 raft_consensus.cc:491] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:51.787523 13961 pending_rounds.cc:85] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Aborting all ops after (but not including) 7
I20250623 01:59:51.792452 13961 pending_rounds.cc:107] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20250623 01:59:51.792711 13961 raft_consensus.cc:2887] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:51.795581 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:51.769871 17320 raft_consensus.cc:2747] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250623 01:59:51.723686 17122 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.785987 17133 raft_consensus.cc:2747] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250623 01:59:51.794900 17365 raft_consensus.cc:3058] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Advancing to term 13
I20250623 01:59:51.819968 17122 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:51.840209 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 43702.9us
I20250623 01:59:51.839118 13708 catalog_manager.cc:5582] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } attrs { replace: true } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:51.843516 13926 leader_election.cc:304] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.844805 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:51.845391 13958 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 01:59:51.846339 13804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:51.846908 13804 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:51.848021 13852 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:51.852692 17296 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 01:59:51.850715 17050 raft_consensus.cc:2747] T 2d3844327a1b452da89d70d856d0a80f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:51.849995 17133 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.009s
I20250623 01:59:51.856359 17296 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:51.850270 17379 consensus_queue.cc:1035] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.009s
I20250623 01:59:51.861250 17296 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:51.873394 13882 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:51.874176 13882 raft_consensus.cc:604] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:51.883082 13708 catalog_manager.cc:5582] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 7 to 9, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New cstate: current_term: 8 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:51.884935 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:51.895141 13926 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:51.911125 17365 raft_consensus.cc:513] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.922363 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:51.923436 13809 raft_consensus.cc:2391] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 13: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:51.925030 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20d8c11ac244feeaecb254a76559f76" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:51.927299 13886 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7560a94e265b433dbc427ef13c120669"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:51.928207 13886 raft_consensus.cc:604] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:51.928555 13886 raft_consensus.cc:612] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:51.929808 17050 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:51.948592 13882 raft_consensus.cc:2466] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 13.
I20250623 01:59:51.953250 13925 leader_election.cc:304] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:51.955022 17257 raft_consensus.cc:2802] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 13 FOLLOWER]: Leader election won for term 13
I20250623 01:59:51.957052 17257 raft_consensus.cc:695] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [term 13 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:51.958842 17257 consensus_queue.cc:237] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:51.965487 13882 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:51.966297 13882 raft_consensus.cc:604] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:51.966727 13882 raft_consensus.cc:612] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:51.951812 17365 leader_election.cc:290] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:51.989071 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52432 (request call id 263) took 6950 ms. Trace:
I20250623 01:59:51.990176 13710 rpcz_store.cc:276] 0623 01:59:45.038949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:45.067317 (+ 28368us) service_pool.cc:225] Handling call
0623 01:59:51.988996 (+6921679us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:50.771882 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:50.798078 (+ 26196us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:50.798116 (+ 38us) write_op.cc:435] Acquired schema lock
0623 01:59:50.798132 (+ 16us) tablet.cc:660] Decoding operations
0623 01:59:50.799074 (+ 942us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:50.799139 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:50.799168 (+ 29us) tablet.cc:683] Acquiring locks for 9 operations
0623 01:59:50.817690 (+ 18522us) tablet.cc:699] Row locks acquired
0623 01:59:50.817710 (+ 20us) write_op.cc:260] PREPARE: finished
0623 01:59:50.817846 (+ 136us) write_op.cc:270] Start()
0623 01:59:50.817981 (+ 135us) write_op.cc:276] Timestamp: P: 1750643990817799 usec, L: 0
0623 01:59:50.818004 (+ 23us) op_driver.cc:348] REPLICATION: starting
0623 01:59:50.818604 (+ 600us) log.cc:844] Serialized 3648 byte log entry
0623 01:59:51.498530 (+679926us) op_driver.cc:464] REPLICATION: finished
0623 01:59:51.723313 (+224783us) write_op.cc:301] APPLY: starting
0623 01:59:51.725891 (+ 2578us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:51.752060 (+ 26169us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:51.752182 (+ 122us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:51.753210 (+ 1028us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:51.753478 (+ 268us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0623 01:59:51.753525 (+ 47us) write_op.cc:312] APPLY: finished
0623 01:59:51.753768 (+ 243us) log.cc:844] Serialized 71 byte log entry
0623 01:59:51.753943 (+ 175us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:51.754066 (+ 123us) write_op.cc:454] Released schema lock
0623 01:59:51.754110 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:51.763406 (+ 9296us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:51.771387 (+ 7981us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":360270,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":202429,"apply.run_cpu_time_us":2806,"apply.run_wall_time_us":33796,"num_ops":9,"prepare.queue_time_us":432391,"prepare.run_cpu_time_us":9029,"prepare.run_wall_time_us":114730,"raft.queue_time_us":340991,"raft.run_cpu_time_us":45959,"raft.run_wall_time_us":139641,"replication_time_us":680374,"thread_start_us":827985,"threads_started":4,"wal-append.queue_time_us":69347}]]}
I20250623 01:59:52.015929 14509 consensus_queue.cc:237] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:52.016683 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 122) took 1806 ms. Trace:
I20250623 01:59:52.001423 17296 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.017064 13708 rpcz_store.cc:276] 0623 01:59:50.235173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:50.271112 (+ 35939us) service_pool.cc:225] Handling call
0623 01:59:52.016610 (+1745498us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:51.985732 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:51.986446 (+ 714us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:51.986498 (+ 52us) write_op.cc:435] Acquired schema lock
0623 01:59:51.986524 (+ 26us) tablet.cc:660] Decoding operations
0623 01:59:51.987051 (+ 527us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:51.987129 (+ 78us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:51.987177 (+ 48us) tablet.cc:683] Acquiring locks for 2 operations
0623 01:59:51.987502 (+ 325us) tablet.cc:699] Row locks acquired
0623 01:59:51.987530 (+ 28us) write_op.cc:260] PREPARE: finished
0623 01:59:51.987716 (+ 186us) write_op.cc:270] Start()
0623 01:59:51.987871 (+ 155us) write_op.cc:276] Timestamp: P: 1750643991987660 usec, L: 0
0623 01:59:51.987901 (+ 30us) op_driver.cc:348] REPLICATION: starting
0623 01:59:51.988980 (+ 1079us) log.cc:844] Serialized 968 byte log entry
0623 01:59:51.991508 (+ 2528us) op_driver.cc:464] REPLICATION: finished
0623 01:59:51.992612 (+ 1104us) write_op.cc:301] APPLY: starting
0623 01:59:51.992690 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:51.992809 (+ 119us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:51.992915 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:51.993331 (+ 416us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:51.993582 (+ 251us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 01:59:51.993646 (+ 64us) write_op.cc:312] APPLY: finished
0623 01:59:51.993869 (+ 223us) log.cc:844] Serialized 29 byte log entry
0623 01:59:51.994050 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:51.994168 (+ 118us) write_op.cc:454] Released schema lock
0623 01:59:51.994220 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:51.994647 (+ 427us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:51.994671 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":51697,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":246,"apply.run_cpu_time_us":1990,"apply.run_wall_time_us":1990,"num_ops":2,"prepare.queue_time_us":85737,"prepare.run_cpu_time_us":3506,"prepare.run_wall_time_us":3636,"replication_time_us":3478,"thread_start_us":58124,"threads_started":1}]]}
I20250623 01:59:52.028384 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet e2928af9e5df4f9eacf59f499ab96dc1 with delete_type TABLET_DATA_TOMBSTONED (TS ee041a8325864e788a383b8356afc652 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58120
I20250623 01:59:52.029217 13808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:52.029912 13808 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:52.021855 13709 catalog_manager.cc:5582] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 7, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 7 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.032455 13709 catalog_manager.cc:5582] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.033758 13709 catalog_manager.cc:5582] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 5, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 5 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.035549 13709 catalog_manager.cc:5582] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 8 to 11, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 11 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:52.036940 13709 catalog_manager.cc:5582] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.040329 13709 catalog_manager.cc:5582] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 9, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 9 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.041596 13709 catalog_manager.cc:5582] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 8, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:52.080785 17296 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:52.085525 13709 catalog_manager.cc:5582] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.087182 13709 catalog_manager.cc:5582] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.089814 17393 tablet_replica.cc:331] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: stopping tablet replica
I20250623 01:59:52.091034 17393 raft_consensus.cc:2241] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:52.093111 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:52.094658 13955 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 8
I20250623 01:59:52.095811 13709 catalog_manager.cc:5582] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.104684 13709 catalog_manager.cc:5582] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.105937 13709 catalog_manager.cc:5582] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 9 to 11, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 11 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.107458 13709 catalog_manager.cc:5582] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:52.108800 13709 catalog_manager.cc:5582] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.109992 13709 catalog_manager.cc:5582] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 3 to 5, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.111284 13955 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:52.111850 13709 catalog_manager.cc:5582] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.112339 13854 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:52.114713 17250 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:52.114017 13709 catalog_manager.cc:5582] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 10, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 10 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.127394 13805 raft_consensus.cc:1273] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250623 01:59:52.127893 17250 raft_consensus.cc:695] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:52.131265 13709 catalog_manager.cc:5582] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 4 to 6, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.132391 17250 consensus_queue.cc:237] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.132833 13709 catalog_manager.cc:5582] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 10, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 10 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.135567 13709 catalog_manager.cc:5582] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 10 to 11, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 11 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.136725 17365 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:59:52.137038 13709 catalog_manager.cc:5582] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 7, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 7 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.140506 17393 raft_consensus.cc:2270] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:52.162103 13882 raft_consensus.cc:1273] T 02e07b6deb3546fcad55d41d2ecd8a5a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250623 01:59:52.171566 13882 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20250623 01:59:52.177208 13806 raft_consensus.cc:1273] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20250623 01:59:52.167503 17257 consensus_queue.cc:1035] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.009s
I20250623 01:59:52.182796 17298 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250623 01:59:52.185623 17198 consensus_queue.cc:1035] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250623 01:59:52.187705 14511 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226"
dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:46348
I20250623 01:59:52.188298 14511 raft_consensus.cc:604] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:52.193001 13808 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:52.195523 13882 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:52.196053 13882 raft_consensus.cc:604] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:52.197468 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:59:52.199102 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:52.199762 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:52.185847 17393 ts_tablet_manager.cc:1905] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:52.217597 13955 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:52.219834 17266 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:52.357435 17379 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:52.358039 17379 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.360378 17379 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 01:59:52.380304 17399 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.128s user 0.004s sys 0.003s
I20250623 01:59:52.422302 13883 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:52.459038 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:52.459774 13805 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:52.474717 13926 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:52.484047 17379 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:52.504451 17379 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:52.524603 17379 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:52.549711 13960 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20250623 01:59:52.553040 13779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 67693.9us
W20250623 01:59:52.495296 17398 log.cc:927] Time spent T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20250623 01:59:52.558262 17122 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:59:52.570889 13805 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20250623 01:59:52.577711 17365 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:52.589517 13708 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 4 to 6, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
W20250623 01:59:52.600044 14478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 183155us
I20250623 01:59:52.608248 17266 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250623 01:59:52.635624 13880 raft_consensus.cc:1273] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 [term 13 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20250623 01:59:52.645623 17302 raft_consensus.cc:491] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c243715fe12445019c33bbeafbaea635)
I20250623 01:59:52.646174 17302 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.648137 17302 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:52.645625 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 258) took 2331 ms. Trace:
I20250623 01:59:52.635059 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:52.672068 13883 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:52.649106 17296 raft_consensus.cc:991] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:52.676945 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:52.661872 13709 rpcz_store.cc:276] 0623 01:59:50.314516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:50.324531 (+ 10015us) service_pool.cc:225] Handling call
0623 01:59:52.645512 (+2320981us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:52.233753 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:52.235113 (+ 1360us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:52.235146 (+ 33us) write_op.cc:435] Acquired schema lock
0623 01:59:52.235172 (+ 26us) tablet.cc:660] Decoding operations
0623 01:59:52.354058 (+118886us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:52.354115 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:52.354153 (+ 38us) tablet.cc:683] Acquiring locks for 21 operations
0623 01:59:52.355824 (+ 1671us) tablet.cc:699] Row locks acquired
0623 01:59:52.355844 (+ 20us) write_op.cc:260] PREPARE: finished
0623 01:59:52.355991 (+ 147us) write_op.cc:270] Start()
0623 01:59:52.356122 (+ 131us) write_op.cc:276] Timestamp: P: 1750643992355946 usec, L: 0
0623 01:59:52.356140 (+ 18us) op_driver.cc:348] REPLICATION: starting
0623 01:59:52.356860 (+ 720us) log.cc:844] Serialized 8168 byte log entry
0623 01:59:52.359565 (+ 2705us) op_driver.cc:464] REPLICATION: finished
0623 01:59:52.360043 (+ 478us) write_op.cc:301] APPLY: starting
0623 01:59:52.360137 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:52.360571 (+ 434us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:52.360685 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:52.362986 (+ 2301us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:52.363380 (+ 394us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=21
0623 01:59:52.363434 (+ 54us) write_op.cc:312] APPLY: finished
0623 01:59:52.396184 (+ 32750us) log.cc:844] Serialized 145 byte log entry
0623 01:59:52.396993 (+ 809us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:52.397190 (+ 197us) write_op.cc:454] Released schema lock
0623 01:59:52.397247 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:52.397647 (+ 400us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:52.397676 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2063,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":262,"apply.run_cpu_time_us":5134,"apply.run_wall_time_us":37571,"num_ops":21,"prepare.queue_time_us":107,"prepare.run_cpu_time_us":9843,"prepare.run_wall_time_us":128816,"replication_time_us":3293}]]}
I20250623 01:59:52.716396 13711 catalog_manager.cc:5582] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 2 to 4, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 4 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:52.721726 17207 raft_consensus.cc:1079] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
W20250623 01:59:52.762331 14481 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 55904.7us
W20250623 01:59:52.769443 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 47285.4us
I20250623 01:59:52.772034 17379 raft_consensus.cc:513] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.773569 13955 raft_consensus.cc:2366] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 10. Current term is 11.
I20250623 01:59:52.776300 17393 ts_tablet_manager.cc:1918] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 8.8
I20250623 01:59:52.776844 17393 log.cc:1199] T e2928af9e5df4f9eacf59f499ab96dc1 P ee041a8325864e788a383b8356afc652: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/wals/e2928af9e5df4f9eacf59f499ab96dc1
I20250623 01:59:52.779543 13956 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:52.764846 17207 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:52.781949 17207 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.783851 17207 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:52.786844 17379 leader_election.cc:290] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:52.789645 17360 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:52.793503 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:52.794123 13962 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
W20250623 01:59:52.803465 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 195931us
I20250623 01:59:52.811214 17266 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:52.811753 17266 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.814306 13805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:52.814911 13805 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:52.818691 17402 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:52.819450 17402 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.807013 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:52.822620 13807 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Committing config change with OpId 12.11: config changed from index 8 to 11, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:52.825132 17403 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:52.825649 17333 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 LEADER]: Committing config change with OpId 12.11: config changed from index 8 to 11, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:52.830617 17412 raft_consensus.cc:991] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:52.833482 17402 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:52.835474 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:52.847043 13960 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:52.841527 13813 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20250623 01:59:52.848516 13813 pending_rounds.cc:85] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: Aborting all ops after (but not including) 4
I20250623 01:59:52.848834 13813 pending_rounds.cc:107] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20250623 01:59:52.801220 13694 catalog_manager.cc:4928] TS ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): tablet e2928af9e5df4f9eacf59f499ab96dc1 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:52.825618 17403 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:52.849633 17403 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
W20250623 01:59:52.849486 13694 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 48388.5us
I20250623 01:59:52.849133 13813 raft_consensus.cc:2887] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 01:59:52.831111 17412 raft_consensus.cc:1079] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:52.853968 13779 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:52.855499 17401 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 01:59:52.855955 17401 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:52.856287 17401 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:52.855326 13810 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:41526
I20250623 01:59:52.857626 13810 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:52.859128 13810 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:52.859998 13880 raft_consensus.cc:2953] T 318e5309795f47f3868e0787ddfdd146 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Committing config change with OpId 12.11: config changed from index 8 to 11, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:52.861701 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:52.869767 13960 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 10.
I20250623 01:59:52.849110 17266 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:52.875161 13779 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:52.876015 17266 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 01:59:52.876396 17266 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 01:59:52.876694 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 234899us
I20250623 01:59:52.877254 17266 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:52.877396 13854 leader_election.cc:400] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 10. Current term is 11.
I20250623 01:59:52.877882 13854 leader_election.cc:403] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20250623 01:59:52.878494 17407 raft_consensus.cc:3058] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 11
I20250623 01:59:52.885810 17296 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:52.887676 17379 consensus_queue.cc:1035] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:52.898183 13811 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:52.900185 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:52.897414 13805 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 01:59:52.904739 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7895c36aec9944b7981b0a0ccacc6556" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:52.908396 13810 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:52.911830 13813 raft_consensus.cc:1238] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Rejecting Update request from peer 7d22741cfb9b408684cb82c58241b2f5 for earlier term 4. Current term is 5. Ops: []
I20250623 01:59:52.912242 17407 raft_consensus.cc:2747] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369) with higher term. Message: Invalid argument: T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 for earlier term 10. Current term is 11.
I20250623 01:59:52.915056 13810 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:52.919046 17369 consensus_queue.cc:1046] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:52.920010 17369 raft_consensus.cc:3053] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Stepping down as leader of term 4
I20250623 01:59:52.920373 17369 raft_consensus.cc:738] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
W20250623 01:59:52.921447 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 115880us
I20250623 01:59:52.921149 17369 consensus_queue.cc:260] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:52.923025 17369 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:52.923153 17302 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:52.923776 17302 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.924207 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:52.924772 13881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:52.925103 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:52.925400 13881 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 4 FOLLOWER]: Advancing to term 5
I20250623 01:59:52.924799 13882 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:52.928697 13926 leader_election.cc:304] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:52.929869 17050 raft_consensus.cc:2802] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 01:59:52.930356 17050 raft_consensus.cc:695] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [term 11 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:52.931210 17050 consensus_queue.cc:237] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.933187 13881 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:52.937314 13776 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:52.941931 13710 catalog_manager.cc:5582] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 8 to 11, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 11 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:52.934660 13884 raft_consensus.cc:2466] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 01:59:52.935145 17302 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:52.946107 13710 catalog_manager.cc:5582] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 10. New cstate: current_term: 10 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:52.954489 13813 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:52.955430 13813 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:52.955883 13813 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:52.957870 17133 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader election won for term 5
I20250623 01:59:52.958405 17133 raft_consensus.cc:695] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:52.959295 17133 consensus_queue.cc:237] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:52.964159 13813 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.975584 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:52.976261 13961 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 01:59:52.977439 17403 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.979259 13854 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:52.980258 13805 raft_consensus.cc:1238] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
I20250623 01:59:52.981163 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:52.982841 13812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:53.015424 13813 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:53.013072 13812 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 01:59:53.018345 13961 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:53.018960 13961 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:53.021731 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52444 (request call id 276) took 7199 ms. Trace:
I20250623 01:59:52.988613 17302 raft_consensus.cc:991] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:53.027604 13710 catalog_manager.cc:5582] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 5 to 8, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 8 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:53.030810 17207 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:53.031780 13883 raft_consensus.cc:2391] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 11: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:52.982348 17266 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:52.987190 17296 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.049s
I20250623 01:59:52.985270 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:53.042732 17266 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:53.049360 13806 raft_consensus.cc:2391] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee041a8325864e788a383b8356afc652 in current term 11: Already voted for candidate 85a58a834b6b4b0dbe197b3dad87e382 in this term.
I20250623 01:59:53.032721 17207 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:53.052161 17207 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:53.052696 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:53.053308 13958 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:53.022281 13711 rpcz_store.cc:276] 0623 01:59:45.822270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:45.953257 (+130987us) service_pool.cc:225] Handling call
0623 01:59:53.021648 (+7068391us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:52.756936 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:52.767115 (+ 10179us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:52.767149 (+ 34us) write_op.cc:435] Acquired schema lock
0623 01:59:52.767164 (+ 15us) tablet.cc:660] Decoding operations
0623 01:59:52.767503 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:52.767554 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:52.767579 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:59:52.767747 (+ 168us) tablet.cc:699] Row locks acquired
0623 01:59:52.767759 (+ 12us) write_op.cc:260] PREPARE: finished
0623 01:59:52.767887 (+ 128us) write_op.cc:270] Start()
0623 01:59:52.767998 (+ 111us) write_op.cc:276] Timestamp: P: 1750643992767846 usec, L: 0
0623 01:59:52.768013 (+ 15us) op_driver.cc:348] REPLICATION: starting
0623 01:59:52.768579 (+ 566us) log.cc:844] Serialized 535 byte log entry
0623 01:59:52.773082 (+ 4503us) op_driver.cc:464] REPLICATION: finished
0623 01:59:52.774246 (+ 1164us) write_op.cc:301] APPLY: starting
0623 01:59:52.774345 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:52.774461 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:52.774579 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:52.774827 (+ 248us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:52.775080 (+ 253us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:59:52.775149 (+ 69us) write_op.cc:312] APPLY: finished
0623 01:59:52.775355 (+ 206us) log.cc:844] Serialized 23 byte log entry
0623 01:59:52.775567 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:52.775667 (+ 100us) write_op.cc:454] Released schema lock
0623 01:59:52.775715 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:52.776148 (+ 433us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:52.776188 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":423779,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":861,"apply.run_cpu_time_us":1866,"apply.run_wall_time_us":1866,"num_ops":1,"prepare.queue_time_us":433,"prepare.run_cpu_time_us":2392,"prepare.run_wall_time_us":22651,"replication_time_us":4911}]]}
I20250623 01:59:53.058346 17403 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:53.059727 17407 raft_consensus.cc:1079] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:53.060556 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:53.063660 13710 catalog_manager.cc:5582] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:53.072211 13961 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:53.072273 13710 catalog_manager.cc:5582] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 5 to 7, leader changed from c243715fe12445019c33bbeafbaea635 (127.12.87.3) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:53.079493 13710 catalog_manager.cc:5582] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:53.083089 13958 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 11.
I20250623 01:59:53.084802 14511 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 5.
I20250623 01:59:53.083205 13710 catalog_manager.cc:5582] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 5 to 8, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 8 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:53.091403 13779 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:53.092406 13779 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:52.964051 17401 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.113330 13756 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 318e5309795f47f3868e0787ddfdd146 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:59:53.121820 17402 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 01:59:53.122416 17402 raft_consensus.cc:695] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:53.123298 17402 consensus_queue.cc:237] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.111966 17277 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:53.131013 17207 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.136127 13961 raft_consensus.cc:1238] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []
W20250623 01:59:53.143473 13776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 122276us
I20250623 01:59:53.158196 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:53.159443 13811 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:53.160032 13811 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:53.159965 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:53.160588 13811 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:53.160816 13813 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:53.184002 17207 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:53.186121 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:53.186944 13958 raft_consensus.cc:3058] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 9
I20250623 01:59:53.188586 13813 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:53.143837 17407 consensus_queue.cc:1046] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20250623 01:59:53.209583 17407 raft_consensus.cc:3053] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:53.210192 17407 raft_consensus.cc:738] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:53.211194 17407 consensus_queue.cc:260] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.212761 17407 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
W20250623 01:59:53.213405 17122 consensus_queue.cc:1162] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "85a58a834b6b4b0dbe197b3dad87e382" responder_term: 9 status { last_received { term: 8 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20250623 01:59:53.198101 17346 log.cc:927] Time spent T 00000000000000000000000000000000 P bc0f35fa176545c3a95a38f727af3536: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20250623 01:59:53.231665 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:53.233434 17277 raft_consensus.cc:695] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:53.234166 13958 raft_consensus.cc:2466] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 01:59:53.234865 17277 consensus_queue.cc:237] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.237910 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:53.243548 13854 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:53.245517 13888 raft_consensus.cc:2391] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 85a58a834b6b4b0dbe197b3dad87e382 in current term 11: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 01:59:53.236779 13811 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:53.252326 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 92453.3us
I20250623 01:59:53.253823 13958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:53.254480 13958 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:53.254575 13883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:53.255156 13883 raft_consensus.cc:3053] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:53.255442 13883 raft_consensus.cc:738] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:53.252450 13811 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:53.255936 13883 consensus_queue.cc:260] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.257373 17250 raft_consensus.cc:2802] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:53.264503 17401 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:53.265686 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:53.266294 13956 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:53.225586 17286 raft_consensus.cc:491] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7d22741cfb9b408684cb82c58241b2f5)
I20250623 01:59:53.271894 17286 raft_consensus.cc:513] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } attrs { replace: true } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:53.276109 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:53.276670 13888 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:53.262017 13883 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:53.281564 13956 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
I20250623 01:59:53.283388 13779 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:53.301481 17286 leader_election.cc:290] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:53.301167 13811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:53.303005 13888 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 6.
W20250623 01:59:53.300634 13854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.3:33369, user_credentials={real_user=slave}} blocked reactor thread for 45352.3us
I20250623 01:59:53.304250 13883 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:53.303439 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:53.305292 13776 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:53.307046 13927 leader_election.cc:304] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:53.309980 17250 raft_consensus.cc:695] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:53.310958 17250 consensus_queue.cc:237] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.313603 17266 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:53.318519 13852 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:53.319550 17020 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250623 01:59:53.305831 17401 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:53.325548 13958 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:53.326823 13709 catalog_manager.cc:5582] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: config changed from index 8 to 11, VOTER c243715fe12445019c33bbeafbaea635 (127.12.87.3) evicted. New cstate: current_term: 12 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:53.333601 17266 raft_consensus.cc:695] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:53.332270 17286 raft_consensus.cc:2747] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250623 01:59:53.334573 17401 raft_consensus.cc:695] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:53.334561 17266 consensus_queue.cc:237] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.343663 13886 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 9.
I20250623 01:59:53.335624 17401 consensus_queue.cc:237] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:53.354310 13888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "318e5309795f47f3868e0787ddfdd146" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:53.358480 13711 catalog_manager.cc:5582] T e20d8c11ac244feeaecb254a76559f76 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 10 to 13, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 13 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } }
I20250623 01:59:53.372689 13951 tablet_service.cc:1515] Processing DeleteTablet for tablet 318e5309795f47f3868e0787ddfdd146 with delete_type TABLET_DATA_TOMBSTONED (TS c243715fe12445019c33bbeafbaea635 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:37408
I20250623 01:59:53.390326 13711 catalog_manager.cc:5582] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:53.405100 13807 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:53.404136 13711 catalog_manager.cc:5582] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:53.422802 17433 tablet_replica.cc:331] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: stopping tablet replica
I20250623 01:59:53.424141 17433 raft_consensus.cc:2241] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:53.425172 17433 raft_consensus.cc:2270] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:53.446362 13813 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20250623 01:59:53.448568 17425 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:53.461678 13886 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250623 01:59:53.463276 17266 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:53.469568 13888 raft_consensus.cc:1273] T 7895c36aec9944b7981b0a0ccacc6556 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20250623 01:59:53.471201 17365 consensus_queue.cc:1035] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:53.476408 14511 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250623 01:59:53.477792 17401 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20250623 01:59:53.511293 14509 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250623 01:59:53.509793 13708 catalog_manager.cc:5582] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:53.524622 17433 ts_tablet_manager.cc:1905] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:53.537477 13711 catalog_manager.cc:5582] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 4 to 5, leader changed from 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 5 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:53.543632 17277 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:53.552289 13958 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250623 01:59:53.576339 17433 ts_tablet_manager.cc:1918] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 12.10
I20250623 01:59:53.576876 17433 log.cc:1199] T 318e5309795f47f3868e0787ddfdd146 P c243715fe12445019c33bbeafbaea635: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-2-root/wals/318e5309795f47f3868e0787ddfdd146
I20250623 01:59:53.579078 17266 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:53.584908 13697 catalog_manager.cc:4928] TS c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): tablet 318e5309795f47f3868e0787ddfdd146 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:53.592932 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:53.626657 13709 catalog_manager.cc:5582] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 9, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 9 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:53.660657 13956 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20250623 01:59:53.663017 17401 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:53.664472 17401 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Committing config change with OpId 6.8: config changed from index 5 to 8, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:53.666127 13956 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 5 to 8, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:53.681834 14511 raft_consensus.cc:2953] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 5 to 8, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:53.700057 13886 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20250623 01:59:53.702673 17277 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:53.720877 13695 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f023b481774a4d06bf5c5c62a3df3012 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:59:54.551123 13883 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:54.552636 13883 raft_consensus.cc:604] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Received request to transfer leadership to TS 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:54.555522 13880 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:54.556099 13880 raft_consensus.cc:604] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:54.556435 13880 raft_consensus.cc:612] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:59:54.560267 13880 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:54.562443 13880 raft_consensus.cc:604] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:54.568431 13710 catalog_manager.cc:5582] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 8 to 9, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 9 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:54.569646 13879 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:54.570206 13879 raft_consensus.cc:604] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:54.573163 13708 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52444 (request call id 277) took 1374 ms. Trace:
I20250623 01:59:54.573886 13879 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:54.574540 13879 raft_consensus.cc:604] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:54.570346 13710 catalog_manager.cc:5582] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:54.578799 13710 catalog_manager.cc:5582] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:54.580363 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250623 01:59:54.580199 13710 catalog_manager.cc:5582] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 8 to 9, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 9 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:54.582384 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:54.573516 13708 rpcz_store.cc:276] 0623 01:59:53.199080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:53.199788 (+ 708us) service_pool.cc:225] Handling call
0623 01:59:54.573106 (+1373318us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:53.533204 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:53.533777 (+ 573us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:53.533812 (+ 35us) write_op.cc:435] Acquired schema lock
0623 01:59:53.533832 (+ 20us) tablet.cc:660] Decoding operations
0623 01:59:53.534204 (+ 372us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:53.534256 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:53.534282 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:59:53.534467 (+ 185us) tablet.cc:699] Row locks acquired
0623 01:59:53.534486 (+ 19us) write_op.cc:260] PREPARE: finished
0623 01:59:53.534625 (+ 139us) write_op.cc:270] Start()
0623 01:59:53.534728 (+ 103us) write_op.cc:276] Timestamp: P: 1750643993534579 usec, L: 0
0623 01:59:53.534748 (+ 20us) op_driver.cc:348] REPLICATION: starting
0623 01:59:53.535364 (+ 616us) log.cc:844] Serialized 535 byte log entry
0623 01:59:53.539172 (+ 3808us) op_driver.cc:464] REPLICATION: finished
0623 01:59:53.539602 (+ 430us) write_op.cc:301] APPLY: starting
0623 01:59:53.539678 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:53.539775 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:53.539875 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:53.540710 (+ 835us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:53.542851 (+ 2141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:59:53.545507 (+ 2656us) write_op.cc:312] APPLY: finished
0623 01:59:53.545702 (+ 195us) log.cc:844] Serialized 23 byte log entry
0623 01:59:53.545853 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:53.545950 (+ 97us) write_op.cc:454] Released schema lock
0623 01:59:53.545987 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:53.546365 (+ 378us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:53.546393 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":858686,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":224,"apply.run_cpu_time_us":1711,"apply.run_wall_time_us":6738,"num_ops":1,"prepare.queue_time_us":96,"prepare.run_cpu_time_us":2495,"prepare.run_wall_time_us":7135,"replication_time_us":4295,"spinlock_wait_cycles":6400}]]}
I20250623 01:59:54.590866 13958 raft_consensus.cc:1273] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:54.597563 17422 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250623 01:59:54.627737 13887 raft_consensus.cc:1273] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250623 01:59:54.629122 13709 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52438 (request call id 260) took 1068 ms. Trace:
I20250623 01:59:54.629113 17422 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250623 01:59:54.629571 13709 rpcz_store.cc:276] 0623 01:59:53.560351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:53.560488 (+ 137us) service_pool.cc:225] Handling call
0623 01:59:54.629061 (+1068573us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:53.650843 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:53.651532 (+ 689us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:53.651576 (+ 44us) write_op.cc:435] Acquired schema lock
0623 01:59:53.651599 (+ 23us) tablet.cc:660] Decoding operations
0623 01:59:53.652006 (+ 407us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:53.652067 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:53.652108 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0623 01:59:53.652344 (+ 236us) tablet.cc:699] Row locks acquired
0623 01:59:53.652376 (+ 32us) write_op.cc:260] PREPARE: finished
0623 01:59:53.652577 (+ 201us) write_op.cc:270] Start()
0623 01:59:53.652699 (+ 122us) write_op.cc:276] Timestamp: P: 1750643993652533 usec, L: 0
0623 01:59:53.652732 (+ 33us) op_driver.cc:348] REPLICATION: starting
0623 01:59:53.674982 (+ 22250us) log.cc:844] Serialized 535 byte log entry
0623 01:59:53.677095 (+ 2113us) op_driver.cc:464] REPLICATION: finished
0623 01:59:53.677484 (+ 389us) write_op.cc:301] APPLY: starting
0623 01:59:53.677570 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:53.677664 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:53.677767 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:53.677952 (+ 185us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:53.678140 (+ 188us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 01:59:53.678198 (+ 58us) write_op.cc:312] APPLY: finished
0623 01:59:53.678371 (+ 173us) log.cc:844] Serialized 23 byte log entry
0623 01:59:53.678526 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:53.678614 (+ 88us) write_op.cc:454] Released schema lock
0623 01:59:53.678658 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:53.679024 (+ 366us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:53.679058 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6644,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":184,"apply.run_cpu_time_us":1501,"apply.run_wall_time_us":1501,"num_ops":1,"prepare.queue_time_us":5617,"prepare.run_cpu_time_us":3085,"prepare.run_wall_time_us":25018,"replication_time_us":24196}]]}
I20250623 01:59:54.586062 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:54.661562 17410 raft_consensus.cc:991] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:54.662189 17410 raft_consensus.cc:1079] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:54.683871 13951 tablet_service.cc:1515] Processing DeleteTablet for tablet 318e5309795f47f3868e0787ddfdd146 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 11)) from {username='slave'} at 127.0.0.1:37408
I20250623 01:59:54.661144 13960 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250623 01:59:54.696522 13882 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250623 01:59:54.697362 17402 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:54.704102 17424 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:54.707227 17360 raft_consensus.cc:991] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:54.707752 17360 raft_consensus.cc:991] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:54.708061 17245 raft_consensus.cc:1079] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:54.710368 13886 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250623 01:59:54.708173 17360 raft_consensus.cc:1079] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:54.715029 17402 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:54.697661 13957 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250623 01:59:54.723883 17424 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:54.725423 13956 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:54.725971 13962 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:54.725484 13960 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:54.726681 13962 raft_consensus.cc:491] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:54.726936 13960 raft_consensus.cc:491] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:54.727180 13962 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:54.727392 13960 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:54.730178 13955 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250623 01:59:54.726023 13956 raft_consensus.cc:491] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:54.731673 13956 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:54.737949 17122 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:54.738538 17122 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } attrs { replace: true } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:54.742300 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:54.742740 17122 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:54.750931 13956 raft_consensus.cc:513] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.751284 13813 raft_consensus.cc:1273] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250623 01:59:54.749627 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:54.752578 13962 raft_consensus.cc:513] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.764299 13810 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:54.763499 13956 leader_election.cc:290] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:54.765043 13810 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:54.765848 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:54.767488 13813 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:54.769330 13697 catalog_manager.cc:4928] TS c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369): tablet 318e5309795f47f3868e0787ddfdd146 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:54.765098 13962 leader_election.cc:290] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:54.774489 13854 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635
I20250623 01:59:54.779286 17303 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:54.790019 13960 raft_consensus.cc:513] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.792567 13807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:54.793126 13807 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:54.761546 17122 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250623 01:59:54.801201 13807 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:54.802359 13926 leader_election.cc:304] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:54.813607 13810 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:54.814035 13960 leader_election.cc:290] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:54.814867 13926 leader_election.cc:304] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:54.814896 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "939deead84d54fe0b858c13a3dfaf686" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:54.815608 13886 raft_consensus.cc:3053] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:54.815984 13886 raft_consensus.cc:738] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:59:54.818143 13742 debug-util.cc:398] Leaking SignalData structure 0x7b0800252940 after lost signal to thread 12639
I20250623 01:59:54.816848 13886 consensus_queue.cc:260] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.819078 13886 raft_consensus.cc:3058] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
W20250623 01:59:54.819260 13742 debug-util.cc:398] Leaking SignalData structure 0x7b080029aec0 after lost signal to thread 13745
W20250623 01:59:54.820513 13742 debug-util.cc:398] Leaking SignalData structure 0x7b08003417c0 after lost signal to thread 13837
W20250623 01:59:54.820756 13742 debug-util.cc:398] Leaking SignalData structure 0x7b08003a3ac0 after lost signal to thread 13838
I20250623 01:59:54.821267 17050 raft_consensus.cc:2802] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election won for term 10
W20250623 01:59:54.821817 13742 debug-util.cc:398] Leaking SignalData structure 0x7b080031dfa0 after lost signal to thread 13912
W20250623 01:59:54.823356 13742 debug-util.cc:398] Leaking SignalData structure 0x7b080032d5e0 after lost signal to thread 13986
I20250623 01:59:54.825489 17425 raft_consensus.cc:2802] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:54.826110 17425 raft_consensus.cc:695] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:54.826910 17425 consensus_queue.cc:237] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.827492 13813 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
W20250623 01:59:54.827661 13742 debug-util.cc:398] Leaking SignalData structure 0x7b0800269d80 after lost signal to thread 14540
I20250623 01:59:54.828305 13886 raft_consensus.cc:2466] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:54.828671 13926 leader_election.cc:304] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
W20250623 01:59:54.828014 13742 debug-util.cc:398] Leaking SignalData structure 0x7b080023ad00 after lost signal to thread 14541
I20250623 01:59:54.829599 17365 raft_consensus.cc:2802] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 01:59:54.830245 17365 raft_consensus.cc:695] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:54.831259 17365 consensus_queue.cc:237] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.833451 17050 raft_consensus.cc:695] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:54.834628 17050 consensus_queue.cc:237] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.846616 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f13ab3fda6f4c6b9b68d33e7a89c988" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:54.847241 13886 raft_consensus.cc:3053] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Stepping down as leader of term 7
I20250623 01:59:54.847375 13813 raft_consensus.cc:1238] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:54.847586 13886 raft_consensus.cc:738] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:54.852178 13810 raft_consensus.cc:1238] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 7. Current term is 8. Ops: []
I20250623 01:59:54.853895 13813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382" is_pre_election: true
I20250623 01:59:54.858069 13886 consensus_queue.cc:260] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.859133 13886 raft_consensus.cc:3058] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 01:59:54.872807 13708 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 reported cstate change: config changed from index 5 to 8, VOTER ee041a8325864e788a383b8356afc652 (127.12.87.2) evicted. New cstate: current_term: 6 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } } }
I20250623 01:59:54.877853 13711 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38034 (request call id 127) took 1229 ms. Trace:
I20250623 01:59:54.878265 13711 rpcz_store.cc:276] 0623 01:59:53.648860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:53.649018 (+ 158us) service_pool.cc:225] Handling call
0623 01:59:54.877782 (+1228764us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":822193,"reported_tablets":1}
I20250623 01:59:54.888160 13710 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52432 (request call id 265) took 1464 ms. Trace:
I20250623 01:59:54.911720 13886 raft_consensus.cc:2466] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 01:59:54.913456 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet f023b481774a4d06bf5c5c62a3df3012 with delete_type TABLET_DATA_TOMBSTONED (TS ee041a8325864e788a383b8356afc652 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:58120
W20250623 01:59:54.915480 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 56649.5us
I20250623 01:59:54.855211 17277 raft_consensus.cc:491] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee041a8325864e788a383b8356afc652)
I20250623 01:59:54.888486 13710 rpcz_store.cc:276] 0623 01:59:53.423655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 01:59:53.423826 (+ 171us) service_pool.cc:225] Handling call
0623 01:59:54.888102 (+1464276us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 01:59:54.632572 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 01:59:54.633106 (+ 534us) write_op.cc:432] Acquiring schema lock in shared mode
0623 01:59:54.633142 (+ 36us) write_op.cc:435] Acquired schema lock
0623 01:59:54.633154 (+ 12us) tablet.cc:660] Decoding operations
0623 01:59:54.633739 (+ 585us) write_op.cc:620] Acquiring the partition lock for write op
0623 01:59:54.633804 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0623 01:59:54.633845 (+ 41us) tablet.cc:683] Acquiring locks for 4 operations
0623 01:59:54.634221 (+ 376us) tablet.cc:699] Row locks acquired
0623 01:59:54.634234 (+ 13us) write_op.cc:260] PREPARE: finished
0623 01:59:54.634379 (+ 145us) write_op.cc:270] Start()
0623 01:59:54.634495 (+ 116us) write_op.cc:276] Timestamp: P: 1750643994634326 usec, L: 0
0623 01:59:54.634517 (+ 22us) op_driver.cc:348] REPLICATION: starting
0623 01:59:54.635107 (+ 590us) log.cc:844] Serialized 1682 byte log entry
0623 01:59:54.680257 (+ 45150us) op_driver.cc:464] REPLICATION: finished
0623 01:59:54.859897 (+179640us) write_op.cc:301] APPLY: starting
0623 01:59:54.859982 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0623 01:59:54.860131 (+ 149us) tablet.cc:1368] finished BulkCheckPresence
0623 01:59:54.860245 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 01:59:54.860718 (+ 473us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 01:59:54.860937 (+ 219us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0623 01:59:54.860986 (+ 49us) write_op.cc:312] APPLY: finished
0623 01:59:54.861210 (+ 224us) log.cc:844] Serialized 41 byte log entry
0623 01:59:54.861402 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:54.861493 (+ 91us) write_op.cc:454] Released schema lock
0623 01:59:54.861542 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0623 01:59:54.861872 (+ 330us) write_op.cc:489] Releasing partition, row and schema locks
0623 01:59:54.861909 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":887204,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":179416,"apply.run_cpu_time_us":1967,"apply.run_wall_time_us":1967,"num_ops":4,"prepare.queue_time_us":834,"prepare.run_cpu_time_us":2835,"prepare.run_wall_time_us":11648,"replication_time_us":45597,"thread_start_us":169983,"threads_started":2}]]}
I20250623 01:59:54.953903 17475 tablet_replica.cc:331] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: stopping tablet replica
I20250623 01:59:54.939929 17277 raft_consensus.cc:513] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.961812 17277 leader_election.cc:290] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:54.969523 17319 raft_consensus.cc:491] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:54.970109 17319 raft_consensus.cc:513] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:54.972172 17319 leader_election.cc:290] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:54.975454 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:54.977602 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:54.983408 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5ada8676314a73af7cef4c5cd5553e" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:54.994364 17245 consensus_queue.cc:1035] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:54.997493 13962 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9e5783d0e1d49b899145b9d9e01f6f9" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:55.023150 17475 raft_consensus.cc:2241] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:55.023895 17475 raft_consensus.cc:2270] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:55.026490 13779 leader_election.cc:304] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:55.027807 17414 raft_consensus.cc:2747] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:55.028021 13779 leader_election.cc:304] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382; no voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652
I20250623 01:59:55.028825 17475 ts_tablet_manager.cc:1905] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 01:59:55.034984 17414 raft_consensus.cc:2747] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250623 01:59:55.043757 17423 raft_consensus.cc:991] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652: : Instructing follower 85a58a834b6b4b0dbe197b3dad87e382 to start an election
I20250623 01:59:55.044273 17423 raft_consensus.cc:1079] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Signalling peer 85a58a834b6b4b0dbe197b3dad87e382 to start an election
W20250623 01:59:55.093674 13852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 61450.3us
I20250623 01:59:55.103662 13806 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3"
dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
from {username='slave'} at 127.0.0.1:48572
I20250623 01:59:55.104257 13806 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:55.104625 13806 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:55.111915 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd540ef429946c987f593ab36327b5a" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:55.112594 13885 raft_consensus.cc:3053] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Stepping down as leader of term 9
I20250623 01:59:55.112895 13885 raft_consensus.cc:738] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:55.115082 13885 consensus_queue.cc:260] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:55.127424 13809 raft_consensus.cc:1238] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Rejecting Update request from peer ee041a8325864e788a383b8356afc652 for earlier term 9. Current term is 10. Ops: []
I20250623 01:59:55.136081 13885 raft_consensus.cc:3058] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 01:59:55.135331 13806 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:55.142258 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:55.142722 13887 raft_consensus.cc:3053] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Stepping down as leader of term 11
I20250623 01:59:55.142939 13887 raft_consensus.cc:738] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:55.143541 13887 consensus_queue.cc:260] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:55.144613 13887 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:55.154865 13887 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 12.
I20250623 01:59:55.161284 13776 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:55.162184 17424 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Leader election won for term 12
I20250623 01:59:55.162719 17424 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 LEADER]: Becoming Leader. State: Replica: 85a58a834b6b4b0dbe197b3dad87e382, State: Running, Role: LEADER
I20250623 01:59:55.164597 17424 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:55.169968 13806 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [CANDIDATE]: Term 12 election: Requested vote from peers ee041a8325864e788a383b8356afc652 (127.12.87.2:43055), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:55.171447 14509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "85a58a834b6b4b0dbe197b3dad87e382" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:55.172024 14509 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:55.181000 14509 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a58a834b6b4b0dbe197b3dad87e382 in term 12.
W20250623 01:59:55.189679 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 58969.1us
I20250623 01:59:55.228626 17475 ts_tablet_manager.cc:1918] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.7
I20250623 01:59:55.229182 17475 log.cc:1199] T f023b481774a4d06bf5c5c62a3df3012 P ee041a8325864e788a383b8356afc652: Deleting WAL directory at /tmp/dist-test-taske7LeWV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750643830703842-12636-0/minicluster-data/ts-1-root/wals/f023b481774a4d06bf5c5c62a3df3012
I20250623 01:59:55.193094 13710 catalog_manager.cc:5582] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 9 to 10, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 10 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:55.246263 13710 catalog_manager.cc:5582] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 7 to 8, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 8 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:55.266496 13694 catalog_manager.cc:4928] TS ee041a8325864e788a383b8356afc652 (127.12.87.2:43055): tablet f023b481774a4d06bf5c5c62a3df3012 (table test-workload [id=865003f5f3954d04903f64a7082491b7]) successfully deleted
I20250623 01:59:55.274029 13885 raft_consensus.cc:2466] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 01:59:55.275359 13882 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250623 01:59:55.277820 17454 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:55.279376 13710 catalog_manager.cc:5582] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 7 to 8, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 8 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:55.318524 13810 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:55.320259 17454 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:55.320336 13711 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 reported cstate change: term changed from 11 to 12, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1). New cstate: current_term: 12 leader_uuid: "85a58a834b6b4b0dbe197b3dad87e382" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:55.324909 13806 raft_consensus.cc:1273] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250623 01:59:55.327706 17454 consensus_queue.cc:1035] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:55.341656 13887 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:55.343387 17454 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:55.362644 13810 raft_consensus.cc:1273] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250623 01:59:55.364481 17365 consensus_queue.cc:1035] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:55.368003 13885 raft_consensus.cc:1273] T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250623 01:59:55.370337 17454 consensus_queue.cc:1035] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250623 01:59:55.461944 17486 log.cc:927] Time spent T 939deead84d54fe0b858c13a3dfaf686 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20250623 01:59:55.607502 13882 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20250623 01:59:55.608992 17422 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250623 01:59:55.682704 14509 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Refusing update from remote peer 85a58a834b6b4b0dbe197b3dad87e382: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20250623 01:59:55.684127 17414 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250623 01:59:55.920337 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:56.082466 13887 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "48dd5f33321544e58bc600dbff358693"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:56.083118 13887 raft_consensus.cc:604] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:56.085582 13887 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "933266ad75974c449e00b527ab278f8f"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:56.086110 13887 raft_consensus.cc:604] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:56.088305 13887 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "afad1ff768904b11936d338d33df7998"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:56.088871 13887 raft_consensus.cc:604] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:56.094218 17245 raft_consensus.cc:991] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:56.094669 17245 raft_consensus.cc:1079] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:56.096364 13955 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "48dd5f33321544e58bc600dbff358693"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:56.096915 13955 raft_consensus.cc:491] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:56.097252 13955 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:56.103475 13887 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4"
dest_uuid: "ee041a8325864e788a383b8356afc652"
mode: GRACEFUL
new_leader_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:50816
I20250623 01:59:56.104104 13887 raft_consensus.cc:604] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Received request to transfer leadership to TS c243715fe12445019c33bbeafbaea635
I20250623 01:59:56.106204 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250623 01:59:56.107748 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:56.108418 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:56.108882 13955 raft_consensus.cc:513] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.118600 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:56.119316 13806 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:56.130484 13806 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 01:59:56.132081 13926 leader_election.cc:304] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:56.132874 17050 raft_consensus.cc:2802] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 01:59:56.132786 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48dd5f33321544e58bc600dbff358693" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:56.133556 17050 raft_consensus.cc:695] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [term 11 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:56.133598 13887 raft_consensus.cc:3053] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:56.134147 13887 raft_consensus.cc:738] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:56.134511 17050 consensus_queue.cc:237] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.134856 13887 consensus_queue.cc:260] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.135864 13887 raft_consensus.cc:3058] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:56.143365 13955 leader_election.cc:290] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:56.148674 13887 raft_consensus.cc:2466] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 01:59:56.165983 13711 catalog_manager.cc:5582] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 10 to 11, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 11 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:56.183416 17406 raft_consensus.cc:991] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:56.183941 17406 raft_consensus.cc:1079] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:56.185876 13957 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "afad1ff768904b11936d338d33df7998"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:56.186511 13957 raft_consensus.cc:491] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:56.186805 13957 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:56.213001 13957 raft_consensus.cc:513] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.249539 13887 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:56.250190 13887 raft_consensus.cc:3053] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Stepping down as leader of term 6
I20250623 01:59:56.250480 13887 raft_consensus.cc:738] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:56.251222 13887 consensus_queue.cc:260] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.252298 13887 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:56.252293 13957 leader_election.cc:290] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:56.312598 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afad1ff768904b11936d338d33df7998" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:56.314262 13806 raft_consensus.cc:3058] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:56.368185 13806 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:56.376332 13926 leader_election.cc:304] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:56.382520 17406 raft_consensus.cc:991] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:56.383399 17406 raft_consensus.cc:1079] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:56.386883 17454 raft_consensus.cc:2802] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 01:59:56.387701 17454 raft_consensus.cc:695] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:56.388607 13960 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:56.393890 13960 raft_consensus.cc:491] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:56.394311 13960 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:56.414300 17454 consensus_queue.cc:237] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 01:59:56.443817 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 67958.2us
I20250623 01:59:56.449128 13960 raft_consensus.cc:513] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.464769 13960 leader_election.cc:290] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:56.465914 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:56.466614 13885 raft_consensus.cc:3053] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Stepping down as leader of term 8
I20250623 01:59:56.466943 13885 raft_consensus.cc:738] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:56.467723 13885 consensus_queue.cc:260] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.475440 13809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfe9e13c26c84532b389e343edf7cef4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:56.476351 13809 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:56.479025 13885 raft_consensus.cc:3058] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:56.496541 13809 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:56.497664 13926 leader_election.cc:304] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 01:59:56.503530 17497 raft_consensus.cc:2802] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:56.504256 17497 raft_consensus.cc:695] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [term 9 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:56.505481 17497 consensus_queue.cc:237] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.511673 13710 catalog_manager.cc:5582] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:56.521777 13882 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20250623 01:59:56.523732 17496 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250623 01:59:56.558715 13887 raft_consensus.cc:2466] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 01:59:56.573293 13885 raft_consensus.cc:2466] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:56.614118 17406 raft_consensus.cc:991] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652: : Instructing follower c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:56.615231 17406 raft_consensus.cc:1079] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Signalling peer c243715fe12445019c33bbeafbaea635 to start an election
I20250623 01:59:56.620620 13711 catalog_manager.cc:5582] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 8 to 9, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 9 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:56.622076 13957 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "933266ad75974c449e00b527ab278f8f"
dest_uuid: "c243715fe12445019c33bbeafbaea635"
from {username='slave'} at 127.0.0.1:37426
I20250623 01:59:56.622653 13957 raft_consensus.cc:491] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:59:56.623070 13957 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:56.643546 13957 raft_consensus.cc:513] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.655094 13806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "85a58a834b6b4b0dbe197b3dad87e382"
I20250623 01:59:56.655686 13806 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:56.655756 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "933266ad75974c449e00b527ab278f8f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:56.656527 13882 raft_consensus.cc:3053] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Stepping down as leader of term 10
I20250623 01:59:56.656850 13882 raft_consensus.cc:738] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 01:59:56.657717 13882 consensus_queue.cc:260] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.658913 13882 raft_consensus.cc:3058] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
I20250623 01:59:56.672326 13957 leader_election.cc:290] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:56.688721 13810 raft_consensus.cc:1273] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20250623 01:59:56.691562 17496 consensus_queue.cc:1035] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 01:59:56.700835 13882 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 01:59:56.717978 13925 leader_election.cc:304] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 01:59:56.718910 17496 raft_consensus.cc:2802] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 01:59:56.719461 17496 raft_consensus.cc:695] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [term 11 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:56.720485 17496 consensus_queue.cc:237] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:56.739107 13806 raft_consensus.cc:2466] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 01:59:56.868808 13711 catalog_manager.cc:5582] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 10 to 11, leader changed from ee041a8325864e788a383b8356afc652 (127.12.87.2) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 11 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:56.887360 13809 raft_consensus.cc:1273] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:56.898504 17454 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:56.954427 13885 raft_consensus.cc:1273] T afad1ff768904b11936d338d33df7998 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 01:59:56.955910 17498 consensus_queue.cc:1035] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250623 01:59:57.053274 13809 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250623 01:59:57.054844 17498 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:57.078637 13886 raft_consensus.cc:1273] T dfe9e13c26c84532b389e343edf7cef4 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250623 01:59:57.082067 17498 consensus_queue.cc:1035] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 01:59:57.340101 13886 raft_consensus.cc:1273] T 933266ad75974c449e00b527ab278f8f P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20250623 01:59:57.348608 13809 raft_consensus.cc:1273] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20250623 01:59:57.350081 17454 consensus_queue.cc:1035] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250623 01:59:57.350910 17496 consensus_queue.cc:1035] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:59:57.477030 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:57.982951 12636 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250623 01:59:57.985275 12636 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250623 01:59:57.986497 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:58.172384 12636 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250623 01:59:58.270390 12636 tablet_server.cc:178] TabletServer@127.12.87.1:0 shutting down...
W20250623 01:59:58.286789 13852 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 11 similar messages]
W20250623 01:59:58.341928 13926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.1:34781, user_credentials={real_user=slave}} blocked reactor thread for 52651us
W20250623 01:59:58.348941 13852 consensus_peers.cc:487] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.350979 13926 consensus_peers.cc:487] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.352309 13926 consensus_peers.cc:487] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.353976 13852 consensus_peers.cc:487] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.357620 13852 consensus_peers.cc:487] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.362865 13852 consensus_peers.cc:487] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.364982 13852 consensus_peers.cc:487] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.385212 13852 consensus_peers.cc:487] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.400565 13926 consensus_peers.cc:487] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.420595 14479 consensus_peers.cc:487] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.422612 14479 consensus_peers.cc:487] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.431624 13852 consensus_peers.cc:487] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.448868 13926 consensus_peers.cc:487] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.450831 13852 consensus_peers.cc:487] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.455533 14479 consensus_peers.cc:487] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.468252 13926 consensus_peers.cc:487] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.471352 13926 consensus_peers.cc:487] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.472108 14479 consensus_peers.cc:487] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.484542 12636 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250623 01:59:58.485739 12636 tablet_replica.cc:331] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.486883 12636 raft_consensus.cc:2241] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.487581 12636 raft_consensus.cc:2270] T bb4438d5ce9e41f3a72f00de14912cff P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.490859 12636 tablet_replica.cc:331] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
W20250623 01:59:58.492973 14479 consensus_peers.cc:487] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.505937 14479 consensus_peers.cc:487] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.507002 12636 raft_consensus.cc:2241] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 LEADER]: Raft consensus shutting down.
W20250623 01:59:58.507247 13926 consensus_peers.cc:487] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.508368 12636 raft_consensus.cc:2270] T 3df3a5b0b6b347b89175eef714fe6a07 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.510972 12636 tablet_replica.cc:331] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.511870 12636 raft_consensus.cc:2241] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
W20250623 01:59:58.512152 13926 consensus_peers.cc:487] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.512462 12636 raft_consensus.cc:2270] T 28d0bf37b0354956a6d195c5259a3d19 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.528375 12636 tablet_replica.cc:331] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.529675 12636 raft_consensus.cc:2241] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.530431 12636 raft_consensus.cc:2270] T 829478b3e1dc4a95a122fcd8be3d2c77 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.533152 12636 tablet_replica.cc:331] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.534034 12636 raft_consensus.cc:2241] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.534603 12636 raft_consensus.cc:2270] T 86045b40aeea4a02993bcd65deecd670 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.537462 12636 tablet_replica.cc:331] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
W20250623 01:59:58.537643 13852 consensus_peers.cc:487] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.538220 12636 raft_consensus.cc:2241] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.539422 12636 raft_consensus.cc:2270] T ba9a759dbf744a06bcb77355efdf4085 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.542095 12636 tablet_replica.cc:331] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
W20250623 01:59:58.542585 13926 consensus_peers.cc:487] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.552496 12636 raft_consensus.cc:2241] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.553193 12636 raft_consensus.cc:2270] T dfe9e13c26c84532b389e343edf7cef4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.555898 12636 tablet_replica.cc:331] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.556674 12636 raft_consensus.cc:2241] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.557263 12636 raft_consensus.cc:2270] T 0c8353de982c4260a5ecf1445df4ba21 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.559863 12636 tablet_replica.cc:331] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.560691 12636 raft_consensus.cc:2241] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.561270 12636 raft_consensus.cc:2270] T f44146f651124d8fba4cc1825f480603 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
W20250623 01:59:58.576376 13852 consensus_peers.cc:487] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.576445 12636 tablet_replica.cc:331] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.582284 12636 raft_consensus.cc:2241] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.582998 12636 raft_consensus.cc:2270] T 5bd540ef429946c987f593ab36327b5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.586040 12636 tablet_replica.cc:331] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.586887 12636 raft_consensus.cc:2241] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.587491 12636 raft_consensus.cc:2270] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.590334 12636 tablet_replica.cc:331] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
W20250623 01:59:58.594024 14479 consensus_peers.cc:487] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.598997 14479 consensus_peers.cc:487] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.605595 12636 raft_consensus.cc:2241] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.606554 12636 raft_consensus.cc:2270] T e2928af9e5df4f9eacf59f499ab96dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
W20250623 01:59:58.612816 14479 consensus_peers.cc:487] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.612970 13926 consensus_peers.cc:487] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.609612 12636 tablet_replica.cc:331] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.620723 12636 raft_consensus.cc:2241] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.621433 12636 raft_consensus.cc:2270] T 0f12346da5a14097bbfa150cb8d1d907 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.624321 12636 tablet_replica.cc:331] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.625196 12636 raft_consensus.cc:2241] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.626351 12636 raft_consensus.cc:2270] T f023b481774a4d06bf5c5c62a3df3012 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.628996 12636 tablet_replica.cc:331] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.642066 12636 raft_consensus.cc:2241] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Raft consensus shutting down.
W20250623 01:59:58.663312 13926 consensus_peers.cc:487] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.663769 14479 consensus_peers.cc:487] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.665891 12636 raft_consensus.cc:2270] T 318e5309795f47f3868e0787ddfdd146 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.669304 12636 tablet_replica.cc:331] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.670220 12636 raft_consensus.cc:2241] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.670759 12636 raft_consensus.cc:2270] T ff8dea72f9f9447abaa5fa803f5e84e8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.680682 12636 tablet_replica.cc:331] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.682309 12636 raft_consensus.cc:2241] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.684190 12636 raft_consensus.cc:2270] T b9e5783d0e1d49b899145b9d9e01f6f9 P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.689208 12636 tablet_replica.cc:331] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.690141 12636 raft_consensus.cc:2241] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.690778 12636 raft_consensus.cc:2270] T 7895c36aec9944b7981b0a0ccacc6556 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.693720 12636 tablet_replica.cc:331] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.695559 12636 raft_consensus.cc:2241] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.698979 12636 raft_consensus.cc:2270] T 05203153a0ed43df91fde4fe090579a7 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.701923 12636 tablet_replica.cc:331] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.702719 12636 raft_consensus.cc:2241] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.703321 12636 raft_consensus.cc:2270] T cd5ada8676314a73af7cef4c5cd5553e P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus is shut down!
W20250623 01:59:58.707234 13852 consensus_peers.cc:487] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:59:58.722115 13926 consensus_peers.cc:487] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.739496 12636 tablet_replica.cc:331] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
W20250623 01:59:58.753099 13852 consensus_peers.cc:487] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.765691 12636 raft_consensus.cc:2241] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus shutting down.
W20250623 01:59:58.777627 13926 consensus_peers.cc:487] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.783138 12636 raft_consensus.cc:2270] T 939deead84d54fe0b858c13a3dfaf686 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.799031 12636 tablet_replica.cc:331] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.800545 12636 raft_consensus.cc:2241] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.801188 12636 raft_consensus.cc:2270] T c708cd61006343f09750b88b2621eec6 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.806219 12636 tablet_replica.cc:331] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.806751 12636 raft_consensus.cc:2241] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.807541 12636 raft_consensus.cc:2270] T adb1e25eef1743848cf218e294bc9226 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.809788 12636 tablet_replica.cc:331] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.822170 12636 raft_consensus.cc:2241] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.822795 12636 raft_consensus.cc:2270] T 1aa785c3afe64dbcac3b658b49195b8c P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.825925 12636 tablet_replica.cc:331] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.826700 12636 raft_consensus.cc:2241] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.827286 12636 raft_consensus.cc:2270] T 839cb616e68044308dfa7e08eadc1903 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.829983 12636 tablet_replica.cc:331] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.830788 12636 raft_consensus.cc:2241] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.831881 12636 raft_consensus.cc:2270] T 1b3574beac134b2eabbd65a4f54b3ab3 P 85a58a834b6b4b0dbe197b3dad87e382 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.834542 12636 tablet_replica.cc:331] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.835358 12636 raft_consensus.cc:2241] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.836014 12636 raft_consensus.cc:2270] T d2a94c049f124a68ad6678cff7f838ac P 85a58a834b6b4b0dbe197b3dad87e382 [term 4 FOLLOWER]: Raft consensus is shut down!
W20250623 01:59:58.838203 13852 consensus_peers.cc:487] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.839162 12636 tablet_replica.cc:331] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.840032 12636 raft_consensus.cc:2241] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.840615 12636 raft_consensus.cc:2270] T e773c5700b0945d1981f481534952e85 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
W20250623 01:59:58.840939 13852 consensus_peers.cc:487] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.843468 12636 tablet_replica.cc:331] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.844355 12636 raft_consensus.cc:2241] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.845189 12636 raft_consensus.cc:2270] T 272c03e6252d489cbf2fdceb36b05dc1 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.848012 12636 tablet_replica.cc:331] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.848748 12636 raft_consensus.cc:2241] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.849287 12636 raft_consensus.cc:2270] T 2540092c63cb4d0cbe07ae4018f99f46 P 85a58a834b6b4b0dbe197b3dad87e382 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.851804 12636 tablet_replica.cc:331] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.854615 12636 raft_consensus.cc:2241] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.855155 12636 raft_consensus.cc:2270] T 92909e3375c549b5baa0595e1750fa1f P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.857789 12636 tablet_replica.cc:331] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
W20250623 01:59:58.858256 13852 consensus_peers.cc:487] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.858510 12636 raft_consensus.cc:2241] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.859661 12636 raft_consensus.cc:2270] T c078b30fc50b475193ae485fbe10e04c P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Raft consensus is shut down!
W20250623 01:59:58.862803 13926 consensus_peers.cc:487] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:58.867646 12636 tablet_replica.cc:331] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.868429 12636 raft_consensus.cc:2241] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.870728 12636 raft_consensus.cc:2270] T 3828994fa152441a95c5034d002516db P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.873348 12636 tablet_replica.cc:331] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.874094 12636 raft_consensus.cc:2241] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.874644 12636 raft_consensus.cc:2270] T 632bb79cb33a491fa5765cd80357f1a4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.877255 12636 tablet_replica.cc:331] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.878147 12636 raft_consensus.cc:2241] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.880735 12636 raft_consensus.cc:2270] T 819094ffb4314769ab2a06512232fc61 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.911602 12636 tablet_replica.cc:331] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.912683 12636 raft_consensus.cc:2241] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.913352 12636 raft_consensus.cc:2270] T 2f02354fc186482aa0fc28cab32390c5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.917129 12636 tablet_replica.cc:331] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.917980 12636 raft_consensus.cc:2241] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.919366 12636 raft_consensus.cc:2270] T 02e07b6deb3546fcad55d41d2ecd8a5a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.948014 12636 tablet_replica.cc:331] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.948822 12636 raft_consensus.cc:2241] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.950016 12636 raft_consensus.cc:2270] T a71ef18fd45046578f969e8e8905e897 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.952787 12636 tablet_replica.cc:331] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.953648 12636 raft_consensus.cc:2241] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.954284 12636 raft_consensus.cc:2270] T e20d8c11ac244feeaecb254a76559f76 P 85a58a834b6b4b0dbe197b3dad87e382 [term 13 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.957387 12636 tablet_replica.cc:331] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.968225 12636 raft_consensus.cc:2241] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.969224 12636 raft_consensus.cc:2270] T bdaf5d40df984f96b90a4d8ace220223 P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.975011 12636 tablet_replica.cc:331] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.975718 12636 raft_consensus.cc:2241] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 LEADER]: Raft consensus shutting down.
I20250623 01:59:58.985637 12636 raft_consensus.cc:2270] T 50c92eb44c6c480d832145fd7c9c8d3f P 85a58a834b6b4b0dbe197b3dad87e382 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:58.996757 12636 tablet_replica.cc:331] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:58.997702 12636 raft_consensus.cc:2241] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:58.998435 12636 raft_consensus.cc:2270] T 2d3844327a1b452da89d70d856d0a80f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.001410 12636 tablet_replica.cc:331] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.006457 12636 raft_consensus.cc:2241] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 LEADER]: Raft consensus shutting down.
I20250623 01:59:59.007751 12636 raft_consensus.cc:2270] T 9cb699d5420143efa03140964d5aa739 P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.010452 12636 tablet_replica.cc:331] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.011235 12636 raft_consensus.cc:2241] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:59.011806 12636 raft_consensus.cc:2270] T 48dd5f33321544e58bc600dbff358693 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.019351 12636 tablet_replica.cc:331] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.020242 12636 raft_consensus.cc:2241] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:59.020884 12636 raft_consensus.cc:2270] T 5d1d49d7480d4088be0d5ed6baae9489 P 85a58a834b6b4b0dbe197b3dad87e382 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.023677 12636 tablet_replica.cc:331] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.024439 12636 raft_consensus.cc:2241] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:59.025044 12636 raft_consensus.cc:2270] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 85a58a834b6b4b0dbe197b3dad87e382 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.027685 12636 tablet_replica.cc:331] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.028458 12636 raft_consensus.cc:2241] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:59.035640 12636 raft_consensus.cc:2270] T afad1ff768904b11936d338d33df7998 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.043550 12636 tablet_replica.cc:331] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.044425 12636 raft_consensus.cc:2241] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:59.045079 12636 raft_consensus.cc:2270] T 933266ad75974c449e00b527ab278f8f P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.053936 12636 tablet_replica.cc:331] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.054811 12636 raft_consensus.cc:2241] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 LEADER]: Raft consensus shutting down.
I20250623 01:59:59.055959 12636 raft_consensus.cc:2270] T ade11e463c53492f8c17cba4e0500ec4 P 85a58a834b6b4b0dbe197b3dad87e382 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.058741 12636 tablet_replica.cc:331] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.059537 12636 raft_consensus.cc:2241] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 LEADER]: Raft consensus shutting down.
I20250623 01:59:59.061159 12636 raft_consensus.cc:2270] T 7560a94e265b433dbc427ef13c120669 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.063802 12636 tablet_replica.cc:331] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.064545 12636 raft_consensus.cc:2241] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 LEADER]: Raft consensus shutting down.
I20250623 01:59:59.065584 12636 raft_consensus.cc:2270] T 32faa630311f4d8e99b289422fe771f5 P 85a58a834b6b4b0dbe197b3dad87e382 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.098910 12636 tablet_replica.cc:331] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.099807 12636 raft_consensus.cc:2241] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 LEADER]: Raft consensus shutting down.
I20250623 01:59:59.100996 12636 raft_consensus.cc:2270] T 7c4f89b1590c439a9b723110eaad17a8 P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.108057 12636 tablet_replica.cc:331] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.110293 12636 raft_consensus.cc:2241] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:59.110978 12636 raft_consensus.cc:2270] T 5d4cc11489de46dcb7d5f77daefd49af P 85a58a834b6b4b0dbe197b3dad87e382 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.113925 12636 tablet_replica.cc:331] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382: stopping tablet replica
I20250623 01:59:59.114774 12636 raft_consensus.cc:2241] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:59.115432 12636 raft_consensus.cc:2270] T e92cd39500a645e79b368d26acb587ed P 85a58a834b6b4b0dbe197b3dad87e382 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:59.638331 17514 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.638872 17514 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:59.643010 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:59.641120 17514 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 01:59:59.652762 14479 leader_election.cc:336] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.653249 14479 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:59.655556 17509 raft_consensus.cc:2747] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 01:59:59.709355 17526 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.709900 17526 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.712170 17526 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:59.713683 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
W20250623 01:59:59.716353 13852 leader_election.cc:336] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.716779 13852 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 01:59:59.717824 17526 raft_consensus.cc:2747] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250623 01:59:59.758482 17454 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.759050 17454 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.760908 17454 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:59.762204 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:59.782104 13926 leader_election.cc:336] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.782608 13926 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:59.783406 17454 raft_consensus.cc:2747] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 01:59:59.808787 17509 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.809393 17509 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:59.811357 17509 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:59.814639 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:59.819104 14479 leader_election.cc:336] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.819512 14479 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:59.820741 17509 raft_consensus.cc:2747] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 01:59:59.842789 16870 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.843343 16870 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.844920 17454 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.847460 17454 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.849368 17454 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:59.856591 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 01:59:59.857183 13885 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
W20250623 01:59:59.857506 13926 leader_election.cc:336] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
W20250623 01:59:59.858217 13852 leader_election.cc:336] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.858539 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:59.859136 13955 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 11.
I20250623 01:59:59.859210 13925 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:59.860338 17454 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250623 01:59:59.860366 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:59.860867 17454 raft_consensus.cc:491] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.861268 17454 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:59.861903 17527 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250623 01:59:59.862287 17527 raft_consensus.cc:491] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.862603 17527 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Advancing to term 12
I20250623 01:59:59.868680 17522 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.869194 17522 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.871152 17522 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:59.877216 17527 raft_consensus.cc:513] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.878342 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 01:59:59.879215 13926 leader_election.cc:336] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.845152 16870 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:59.881218 13925 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 01:59:59.882236 17522 raft_consensus.cc:2747] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250623 01:59:59.884003 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba9a759dbf744a06bcb77355efdf4085" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 01:59:59.884642 13955 raft_consensus.cc:3058] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Advancing to term 12
W20250623 01:59:59.887524 13852 leader_election.cc:336] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.893872 17454 raft_consensus.cc:513] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.896555 17527 leader_election.cc:290] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:59.897330 13955 raft_consensus.cc:2466] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 12.
I20250623 01:59:59.898722 13854 leader_election.cc:304] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:59.901243 17527 raft_consensus.cc:2802] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election won for term 12
I20250623 01:59:59.902871 17527 raft_consensus.cc:695] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [term 12 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
W20250623 01:59:59.905922 13926 leader_election.cc:336] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.908051 17454 leader_election.cc:290] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 01:59:59.918061 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3df3a5b0b6b347b89175eef714fe6a07" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 01:59:59.918721 13885 raft_consensus.cc:3058] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Advancing to term 9
I20250623 01:59:59.936573 13885 raft_consensus.cc:2466] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
I20250623 01:59:59.903807 17527 consensus_queue.cc:237] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.951521 17454 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.952134 17454 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:59.956151 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 01:59:59.956849 14511 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 01:59:59.953998 17454 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 01:59:59.961033 13926 leader_election.cc:336] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 01:59:59.964061 13927 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:59.965011 17536 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 01:59:59.965457 17536 raft_consensus.cc:491] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.965811 17536 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:59.966431 13925 leader_election.cc:304] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 01:59:59.967446 17454 raft_consensus.cc:2802] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader election won for term 9
I20250623 01:59:59.968720 17454 raft_consensus.cc:695] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [term 9 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 01:59:59.970571 17454 consensus_queue.cc:237] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.974062 17536 raft_consensus.cc:513] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 01:59:59.982728 17535 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.983258 16870 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.983491 17535 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.983702 16870 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.985199 17522 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 01:59:59.985200 17535 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:59.985200 16870 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 01:59:59.985837 17522 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 01:59:59.986544 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:59.987723 17536 leader_election.cc:290] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 01:59:59.988085 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 01:59:59.990396 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f023b481774a4d06bf5c5c62a3df3012" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 01:59:59.990904 14511 raft_consensus.cc:3058] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 01:59:59.988318 17522 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.000257 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 02:00:00.006460 17536 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.006858 14511 raft_consensus.cc:2466] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 02:00:00.007026 17536 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.008162 13927 leader_election.cc:304] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, c243715fe12445019c33bbeafbaea635; no voters:
I20250623 02:00:00.009003 17536 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 02:00:00.009284 13926 leader_election.cc:336] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
W20250623 02:00:00.012619 13926 leader_election.cc:336] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.020870 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 02:00:00.021530 13885 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 6.
I20250623 02:00:00.009037 17522 raft_consensus.cc:2802] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 02:00:00.021575 13925 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 02:00:00.022724 17522 raft_consensus.cc:695] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 02:00:00.023630 17536 raft_consensus.cc:2747] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 02:00:00.017302 13711 catalog_manager.cc:5582] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 8 to 9, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 9 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 02:00:00.023912 17522 consensus_queue.cc:237] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 02:00:00.026607 13852 leader_election.cc:336] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.028183 13852 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
W20250623 02:00:00.032246 13852 leader_election.cc:336] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.032764 13852 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 02:00:00.033115 16870 raft_consensus.cc:2747] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250623 02:00:00.033808 16870 raft_consensus.cc:2747] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250623 02:00:00.036916 13925 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 02:00:00.045742 17454 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.052336 17454 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.051976 17536 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 02:00:00.058962 17536 raft_consensus.cc:491] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.059350 17536 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Advancing to term 7
I20250623 02:00:00.066542 13708 catalog_manager.cc:5582] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 11 to 12, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 12 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:00.078292 17454 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.064754 17527 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.082683 17527 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.082757 17536 raft_consensus.cc:513] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.084859 17527 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 02:00:00.086951 13926 leader_election.cc:336] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.088115 17527 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.088256 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 02:00:00.088660 17527 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 02:00:00.090685 17527 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
W20250623 02:00:00.091943 13926 leader_election.cc:336] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.092084 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
W20250623 02:00:00.092666 13926 leader_election.cc:336] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
W20250623 02:00:00.095789 13852 leader_election.cc:336] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.104755 13852 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 02:00:00.105613 16870 raft_consensus.cc:2747] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250623 02:00:00.100085 17536 leader_election.cc:290] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.096661 17527 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.107882 17527 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.111096 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 02:00:00.111665 13960 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 02:00:00.112803 13854 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 02:00:00.114044 17526 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 02:00:00.114427 17526 raft_consensus.cc:491] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.114789 17526 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
W20250623 02:00:00.120311 13852 leader_election.cc:336] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.120780 13852 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 7d22741cfb9b408684cb82c58241b2f5, 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.121467 16870 raft_consensus.cc:2747] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250623 02:00:00.123099 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
W20250623 02:00:00.123339 13852 leader_election.cc:336] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.123490 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3828994fa152441a95c5034d002516db" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 02:00:00.124048 13882 raft_consensus.cc:3058] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 02:00:00.112798 17527 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.135329 17526 raft_consensus.cc:513] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.136587 13886 raft_consensus.cc:2391] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c243715fe12445019c33bbeafbaea635 in current term 8: Already voted for candidate ee041a8325864e788a383b8356afc652 in this term.
I20250623 02:00:00.137233 13882 raft_consensus.cc:2466] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 02:00:00.138262 17526 leader_election.cc:290] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.138446 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71ef18fd45046578f969e8e8905e897" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 02:00:00.139055 13960 raft_consensus.cc:3058] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 02:00:00.139842 17514 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.140004 17543 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.140122 13925 leader_election.cc:304] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.140388 17514 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 02:00:00.140633 17543 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.141229 17454 raft_consensus.cc:2802] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 02:00:00.143503 17454 raft_consensus.cc:695] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [term 7 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 02:00:00.144253 17454 consensus_queue.cc:237] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 02:00:00.146597 13926 leader_election.cc:336] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
W20250623 02:00:00.146950 14479 leader_election.cc:336] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.148159 13925 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
W20250623 02:00:00.149717 13852 leader_election.cc:336] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.149731 17543 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.154438 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 02:00:00.158686 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 02:00:00.159109 13925 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 02:00:00.161180 17526 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.158707 17514 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.162122 13960 raft_consensus.cc:2466] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 8.
I20250623 02:00:00.163271 13882 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 12.
I20250623 02:00:00.174291 14478 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.175530 17514 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20250623 02:00:00.175983 17514 raft_consensus.cc:491] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.176422 17514 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 12 FOLLOWER]: Advancing to term 13
I20250623 02:00:00.176554 17541 raft_consensus.cc:2747] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 02:00:00.177150 13854 leader_election.cc:304] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.179613 17535 raft_consensus.cc:2802] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election won for term 8
I20250623 02:00:00.180287 17535 raft_consensus.cc:695] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [term 8 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 02:00:00.163450 17543 raft_consensus.cc:2747] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250623 02:00:00.181612 17535 consensus_queue.cc:237] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.163446 17526 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 02:00:00.195765 17526 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 02:00:00.196959 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 02:00:00.197796 14511 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
I20250623 02:00:00.198964 13853 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
W20250623 02:00:00.199589 13852 leader_election.cc:336] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.199811 17535 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 02:00:00.200217 17535 raft_consensus.cc:491] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.200574 17535 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 02:00:00.207078 17514 raft_consensus.cc:513] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 02:00:00.213912 17543 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.214475 17543 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.216567 17543 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.218225 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 02:00:00.218844 13886 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 02:00:00.219759 13925 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 02:00:00.219725 17514 leader_election.cc:290] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 13 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.220583 17543 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250623 02:00:00.220933 17543 raft_consensus.cc:491] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.221336 17543 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Advancing to term 11
I20250623 02:00:00.221069 17535 raft_consensus.cc:513] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 02:00:00.223261 14479 leader_election.cc:336] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.224301 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adb1e25eef1743848cf218e294bc9226" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 02:00:00.224849 14511 raft_consensus.cc:3058] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 5 FOLLOWER]: Advancing to term 6
I20250623 02:00:00.225579 17535 leader_election.cc:290] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 02:00:00.228363 17527 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.228886 17527 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.229904 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b3574beac134b2eabbd65a4f54b3ab3" candidate_uuid: "7d22741cfb9b408684cb82c58241b2f5" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 02:00:00.230492 13886 raft_consensus.cc:3058] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Advancing to term 13
I20250623 02:00:00.234154 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 02:00:00.234738 13957 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 5.
W20250623 02:00:00.235879 13852 leader_election.cc:336] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.235877 13854 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 02:00:00.236852 17527 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.236986 17526 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250623 02:00:00.237465 17526 raft_consensus.cc:491] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.237823 17526 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 5 FOLLOWER]: Advancing to term 6
I20250623 02:00:00.239033 17527 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.239516 17527 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 02:00:00.240582 13852 leader_election.cc:336] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.241613 17527 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.242834 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 02:00:00.247054 17543 raft_consensus.cc:513] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 02:00:00.223313 13926 leader_election.cc:336] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
W20250623 02:00:00.250439 13852 leader_election.cc:336] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.250859 13852 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382, c243715fe12445019c33bbeafbaea635
I20250623 02:00:00.252394 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdaf5d40df984f96b90a4d8ace220223" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
W20250623 02:00:00.252542 13926 leader_election.cc:336] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.252633 17527 raft_consensus.cc:2747] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250623 02:00:00.252929 13882 raft_consensus.cc:3058] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Advancing to term 11
I20250623 02:00:00.257905 13882 raft_consensus.cc:2466] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 11.
I20250623 02:00:00.260829 13925 leader_election.cc:304] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.261531 17454 raft_consensus.cc:2802] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader election won for term 11
I20250623 02:00:00.261968 17454 raft_consensus.cc:695] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [term 11 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 02:00:00.262670 17454 consensus_queue.cc:237] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.265446 17541 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.265938 17541 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.267751 17541 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.268321 14511 raft_consensus.cc:2466] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 02:00:00.270769 13853 leader_election.cc:304] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.265446 13886 raft_consensus.cc:2466] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d22741cfb9b408684cb82c58241b2f5 in term 13.
I20250623 02:00:00.270843 17526 raft_consensus.cc:513] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.272684 17535 raft_consensus.cc:2802] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
W20250623 02:00:00.272732 13926 leader_election.cc:336] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.273942 17535 raft_consensus.cc:695] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 02:00:00.274175 14478 leader_election.cc:304] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.275236 17514 raft_consensus.cc:2802] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 13 FOLLOWER]: Leader election won for term 13
I20250623 02:00:00.275750 17514 raft_consensus.cc:695] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [term 13 LEADER]: Becoming Leader. State: Replica: 7d22741cfb9b408684cb82c58241b2f5, State: Running, Role: LEADER
I20250623 02:00:00.276571 17514 consensus_queue.cc:237] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 02:00:00.274866 17535 consensus_queue.cc:237] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 02:00:00.273303 17543 leader_election.cc:290] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 11 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.283392 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 02:00:00.289841 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c4f89b1590c439a9b723110eaad17a8" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 02:00:00.290378 13957 raft_consensus.cc:3058] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 5 FOLLOWER]: Advancing to term 6
I20250623 02:00:00.298664 17527 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.299239 17527 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.309538 13925 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c243715fe12445019c33bbeafbaea635; no voters: 85a58a834b6b4b0dbe197b3dad87e382, ee041a8325864e788a383b8356afc652
I20250623 02:00:00.308660 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 02:00:00.310323 13960 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 9.
I20250623 02:00:00.301106 17527 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.313680 17543 raft_consensus.cc:2747] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250623 02:00:00.310957 17526 leader_election.cc:290] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.315193 13854 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 02:00:00.315939 17526 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 02:00:00.316322 17526 raft_consensus.cc:491] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.316700 17526 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
I20250623 02:00:00.317850 17527 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.318382 17527 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 02:00:00.319631 13852 leader_election.cc:336] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.320482 17527 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
W20250623 02:00:00.320937 13852 leader_election.cc:336] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.322384 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 02:00:00.323174 13956 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 11.
I20250623 02:00:00.324982 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 02:00:00.326885 16870 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250623 02:00:00.327208 16870 raft_consensus.cc:491] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.327558 16870 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Advancing to term 12
W20250623 02:00:00.334807 13852 leader_election.cc:336] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.353744 13957 raft_consensus.cc:2466] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 02:00:00.354979 13854 leader_election.cc:304] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.356110 17527 raft_consensus.cc:2802] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 02:00:00.356958 16870 raft_consensus.cc:513] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.367164 16870 leader_election.cc:290] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.368750 13957 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7560a94e265b433dbc427ef13c120669" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 02:00:00.369215 13957 raft_consensus.cc:3058] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Advancing to term 12
W20250623 02:00:00.375236 13852 leader_election.cc:336] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.383483 13711 catalog_manager.cc:5582] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:00.380888 17526 raft_consensus.cc:513] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.392206 13711 catalog_manager.cc:5582] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 6 to 7, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 7 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 02:00:00.393893 13956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c078b30fc50b475193ae485fbe10e04c" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 02:00:00.394474 13956 raft_consensus.cc:3058] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 02:00:00.394896 13711 catalog_manager.cc:5582] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 10 to 11, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 11 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 02:00:00.387523 17527 raft_consensus.cc:695] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [term 6 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 02:00:00.388185 17454 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.390080 13710 catalog_manager.cc:5582] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 7 to 8, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 8 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:00.386744 13708 catalog_manager.cc:5582] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 reported cstate change: term changed from 12 to 13, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4). New cstate: current_term: 13 leader_uuid: "7d22741cfb9b408684cb82c58241b2f5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 02:00:00.392962 17526 leader_election.cc:290] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.406395 17454 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 02:00:00.403264 12636 thread.cc:535] Waited for 1000ms trying to join with diag-logger (tid 13834)
I20250623 02:00:00.410432 17454 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 02:00:00.410439 13852 leader_election.cc:336] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.401695 13956 raft_consensus.cc:2466] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 10.
W20250623 02:00:00.415621 13926 leader_election.cc:336] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.416314 13854 leader_election.cc:304] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.417179 17526 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.417810 17526 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 02:00:00.417519 13710 catalog_manager.cc:5582] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 6, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:00.420634 17526 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 02:00:00.421205 17526 raft_consensus.cc:2802] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 02:00:00.421696 17526 raft_consensus.cc:695] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [term 10 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 02:00:00.421839 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5" is_pre_election: true
I20250623 02:00:00.422456 14511 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 6.
I20250623 02:00:00.422637 17526 consensus_queue.cc:237] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.423475 13853 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters:
I20250623 02:00:00.424187 16870 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250623 02:00:00.424505 16870 raft_consensus.cc:491] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.424813 16870 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 6 FOLLOWER]: Advancing to term 7
I20250623 02:00:00.428145 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 02:00:00.428718 13882 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 7.
I20250623 02:00:00.429840 13925 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.430788 17454 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250623 02:00:00.431116 17454 raft_consensus.cc:491] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.431511 17454 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 7 FOLLOWER]: Advancing to term 8
I20250623 02:00:00.435250 16870 raft_consensus.cc:513] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
I20250623 02:00:00.436488 17527 consensus_queue.cc:237] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.438174 14511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb699d5420143efa03140964d5aa739" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7d22741cfb9b408684cb82c58241b2f5"
I20250623 02:00:00.440006 14511 raft_consensus.cc:3058] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Advancing to term 7
I20250623 02:00:00.441383 16870 leader_election.cc:290] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), 7d22741cfb9b408684cb82c58241b2f5 (127.12.87.4:39393)
I20250623 02:00:00.449430 14509 raft_consensus.cc:1273] T f023b481774a4d06bf5c5c62a3df3012 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250623 02:00:00.451345 17454 raft_consensus.cc:513] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.451747 17543 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.452476 17543 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 02:00:00.452955 13852 leader_election.cc:336] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.451728 17541 consensus_queue.cc:1035] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250623 02:00:00.456396 17543 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.460352 17454 leader_election.cc:290] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
W20250623 02:00:00.461648 13926 leader_election.cc:336] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
W20250623 02:00:00.462160 13926 leader_election.cc:336] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.464164 13886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade11e463c53492f8c17cba4e0500ec4" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 02:00:00.464726 13886 raft_consensus.cc:3058] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Advancing to term 8
I20250623 02:00:00.465752 13957 raft_consensus.cc:2466] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 12.
I20250623 02:00:00.469436 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652" is_pre_election: true
I20250623 02:00:00.470014 13882 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 9.
W20250623 02:00:00.476047 13852 leader_election.cc:336] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.477497 13886 raft_consensus.cc:2466] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 8.
I20250623 02:00:00.477833 13854 leader_election.cc:304] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.479036 17553 raft_consensus.cc:2802] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election won for term 12
I20250623 02:00:00.479596 17553 raft_consensus.cc:695] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [term 12 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 02:00:00.482789 13925 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.480564 17553 consensus_queue.cc:237] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.483861 17454 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250623 02:00:00.484238 17454 raft_consensus.cc:491] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.484566 17454 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 9 FOLLOWER]: Advancing to term 10
I20250623 02:00:00.486966 16870 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.487402 16870 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.489020 16870 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.490422 13925 leader_election.cc:304] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.491356 17543 raft_consensus.cc:2802] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Leader election won for term 8
W20250623 02:00:00.491468 13926 consensus_peers.cc:487] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 02:00:00.491890 17543 raft_consensus.cc:695] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [term 8 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 02:00:00.490954 13960 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635" is_pre_election: true
I20250623 02:00:00.492619 13960 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 11.
I20250623 02:00:00.492733 17543 consensus_queue.cc:237] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.493598 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters:
I20250623 02:00:00.496479 17526 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250623 02:00:00.496892 17526 raft_consensus.cc:491] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 85a58a834b6b4b0dbe197b3dad87e382)
I20250623 02:00:00.497277 17526 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Advancing to term 12
I20250623 02:00:00.503110 14511 raft_consensus.cc:2466] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 7.
I20250623 02:00:00.504315 13853 leader_election.cc:304] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d22741cfb9b408684cb82c58241b2f5, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.505342 16870 raft_consensus.cc:2802] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Leader election won for term 7
I20250623 02:00:00.505777 16870 raft_consensus.cc:695] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [term 7 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 02:00:00.506534 16870 consensus_queue.cc:237] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 02:00:00.513392 17554 log.cc:927] Time spent T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20250623 02:00:00.530455 13955 raft_consensus.cc:1273] T ba9a759dbf744a06bcb77355efdf4085 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20250623 02:00:00.531901 17535 consensus_queue.cc:1035] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20250623 02:00:00.539863 13926 consensus_peers.cc:487] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:00.554265 13926 consensus_peers.cc:487] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:00.543340 13852 consensus_peers.cc:487] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:00.572849 13852 leader_election.cc:336] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.577491 17454 raft_consensus.cc:513] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.585063 13882 raft_consensus.cc:1273] T 3df3a5b0b6b347b89175eef714fe6a07 P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20250623 02:00:00.573158 13708 catalog_manager.cc:5582] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 5 to 6, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 6 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:00.589125 13708 catalog_manager.cc:5582] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 9 to 10, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 10 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
W20250623 02:00:00.590384 17556 log.cc:927] Time spent T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250623 02:00:00.594842 13711 catalog_manager.cc:5582] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 7 to 8, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 8 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
I20250623 02:00:00.620779 17454 leader_election.cc:290] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), ee041a8325864e788a383b8356afc652 (127.12.87.2:43055)
I20250623 02:00:00.624493 17454 consensus_queue.cc:1035] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250623 02:00:00.639827 13926 leader_election.cc:336] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.642585 17526 raft_consensus.cc:513] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
W20250623 02:00:00.677043 13925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.87.2:43055, user_credentials={real_user=slave}} blocked reactor thread for 80991.9us
W20250623 02:00:00.677335 14479 consensus_peers.cc:487] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:00.679402 13882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c92eb44c6c480d832145fd7c9c8d3f" candidate_uuid: "c243715fe12445019c33bbeafbaea635" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ee041a8325864e788a383b8356afc652"
I20250623 02:00:00.679981 13882 raft_consensus.cc:3058] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 9 FOLLOWER]: Advancing to term 10
W20250623 02:00:00.680605 13852 consensus_peers.cc:487] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 02:00:00.681233 13955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32faa630311f4d8e99b289422fe771f5" candidate_uuid: "ee041a8325864e788a383b8356afc652" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "c243715fe12445019c33bbeafbaea635"
I20250623 02:00:00.681840 13955 raft_consensus.cc:3058] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 11 FOLLOWER]: Advancing to term 12
W20250623 02:00:00.684770 14479 consensus_peers.cc:487] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:00.685189 13852 leader_election.cc:336] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111)
I20250623 02:00:00.687350 17526 leader_election.cc:290] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: Requested vote from peers 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781), c243715fe12445019c33bbeafbaea635 (127.12.87.3:33369)
I20250623 02:00:00.689667 13955 raft_consensus.cc:2466] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee041a8325864e788a383b8356afc652 in term 12.
I20250623 02:00:00.694428 13882 raft_consensus.cc:2466] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c243715fe12445019c33bbeafbaea635 in term 10.
I20250623 02:00:00.690999 13854 leader_election.cc:304] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.701289 13925 leader_election.cc:304] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c243715fe12445019c33bbeafbaea635, ee041a8325864e788a383b8356afc652; no voters: 85a58a834b6b4b0dbe197b3dad87e382
I20250623 02:00:00.701467 16870 raft_consensus.cc:2802] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 12 FOLLOWER]: Leader election won for term 12
I20250623 02:00:00.702054 17454 raft_consensus.cc:2802] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Leader election won for term 10
I20250623 02:00:00.702555 17454 raft_consensus.cc:695] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [term 10 LEADER]: Becoming Leader. State: Replica: c243715fe12445019c33bbeafbaea635, State: Running, Role: LEADER
I20250623 02:00:00.703068 17454 consensus_queue.cc:237] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.702021 16870 raft_consensus.cc:695] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [term 12 LEADER]: Becoming Leader. State: Replica: ee041a8325864e788a383b8356afc652, State: Running, Role: LEADER
I20250623 02:00:00.708088 16870 consensus_queue.cc:237] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } }
I20250623 02:00:00.716037 13955 raft_consensus.cc:1273] T a71ef18fd45046578f969e8e8905e897 P c243715fe12445019c33bbeafbaea635 [term 8 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250623 02:00:00.717767 17526 consensus_queue.cc:1035] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
W20250623 02:00:00.730718 13926 consensus_peers.cc:487] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:00.731350 13882 raft_consensus.cc:1273] T 1b3574beac134b2eabbd65a4f54b3ab3 P ee041a8325864e788a383b8356afc652 [term 13 FOLLOWER]: Refusing update from remote peer 7d22741cfb9b408684cb82c58241b2f5: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20250623 02:00:00.732755 17542 consensus_queue.cc:1035] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20250623 02:00:00.736554 13926 consensus_peers.cc:487] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:00.738972 13852 consensus_peers.cc:487] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 02:00:00.741945 13882 raft_consensus.cc:1273] T 3828994fa152441a95c5034d002516db P ee041a8325864e788a383b8356afc652 [term 7 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250623 02:00:00.745250 17454 consensus_queue.cc:1035] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250623 02:00:00.754995 13852 consensus_peers.cc:487] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:00.766094 14509 raft_consensus.cc:1273] T adb1e25eef1743848cf218e294bc9226 P 7d22741cfb9b408684cb82c58241b2f5 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250623 02:00:00.767997 13852 consensus_peers.cc:487] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:00.767894 16870 consensus_queue.cc:1035] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250623 02:00:00.777277 13711 catalog_manager.cc:5582] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 6 to 7, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 7 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:00.779457 13711 catalog_manager.cc:5582] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 11 to 12, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 12 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:00.780858 13711 catalog_manager.cc:5582] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 reported cstate change: term changed from 11 to 12, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to ee041a8325864e788a383b8356afc652 (127.12.87.2). New cstate: current_term: 12 leader_uuid: "ee041a8325864e788a383b8356afc652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:00.791114 13708 catalog_manager.cc:5582] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 reported cstate change: term changed from 9 to 10, leader changed from 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1) to c243715fe12445019c33bbeafbaea635 (127.12.87.3). New cstate: current_term: 10 leader_uuid: "c243715fe12445019c33bbeafbaea635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a58a834b6b4b0dbe197b3dad87e382" member_type: VOTER last_known_addr { host: "127.12.87.1" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } health_report { overall_health: HEALTHY } } }
W20250623 02:00:00.800402 13852 consensus_peers.cc:487] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:00.831053 14479 consensus_peers.cc:487] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:00.836684 14479 consensus_peers.cc:487] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:00.842029 14479 consensus_peers.cc:487] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:00.859258 13885 raft_consensus.cc:1273] T bdaf5d40df984f96b90a4d8ace220223 P ee041a8325864e788a383b8356afc652 [term 11 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20250623 02:00:00.862026 13852 consensus_peers.cc:487] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:00.865495 17564 consensus_queue.cc:1035] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250623 02:00:00.870469 13852 consensus_peers.cc:487] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:00.889566 13852 consensus_peers.cc:487] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 02:00:00.908856 13957 raft_consensus.cc:1273] T c078b30fc50b475193ae485fbe10e04c P c243715fe12445019c33bbeafbaea635 [term 10 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250623 02:00:00.910310 16870 consensus_queue.cc:1035] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250623 02:00:00.910792 14479 consensus_peers.cc:487] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:00.918860 14479 consensus_peers.cc:487] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:00.934451 13926 consensus_peers.cc:487] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:00.968075 14479 consensus_peers.cc:487] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:00.972426 13852 consensus_peers.cc:487] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:00.972932 13852 consensus_peers.cc:487] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:00.978058 13926 consensus_peers.cc:487] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:01.011596 13955 raft_consensus.cc:1273] T 7560a94e265b433dbc427ef13c120669 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20250623 02:00:01.013134 17526 consensus_queue.cc:1035] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250623 02:00:01.020242 13852 consensus_peers.cc:487] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.024591 13852 consensus_peers.cc:487] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.023636 13926 consensus_peers.cc:487] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.026497 14479 consensus_peers.cc:487] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.028412 13926 consensus_peers.cc:487] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:01.046180 13926 consensus_peers.cc:487] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:01.047258 13882 raft_consensus.cc:1273] T ade11e463c53492f8c17cba4e0500ec4 P ee041a8325864e788a383b8356afc652 [term 8 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20250623 02:00:01.049165 13926 consensus_peers.cc:487] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.052799 13926 consensus_peers.cc:487] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.048098 13852 consensus_peers.cc:487] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:01.061370 17454 consensus_queue.cc:1035] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250623 02:00:01.064482 13926 consensus_peers.cc:487] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.064931 14479 consensus_peers.cc:487] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:01.088755 13955 raft_consensus.cc:1273] T 7c4f89b1590c439a9b723110eaad17a8 P c243715fe12445019c33bbeafbaea635 [term 6 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250623 02:00:01.098026 17526 consensus_queue.cc:1035] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250623 02:00:01.100589 13926 consensus_peers.cc:487] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:01.119966 14509 raft_consensus.cc:1273] T 9cb699d5420143efa03140964d5aa739 P 7d22741cfb9b408684cb82c58241b2f5 [term 7 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250623 02:00:01.125056 16870 consensus_queue.cc:1035] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20250623 02:00:01.140928 13852 consensus_peers.cc:487] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:01.183398 14479 consensus_peers.cc:487] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.201972 13852 consensus_peers.cc:487] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.202670 13852 consensus_peers.cc:487] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 02:00:01.206060 13882 raft_consensus.cc:1273] T 50c92eb44c6c480d832145fd7c9c8d3f P ee041a8325864e788a383b8356afc652 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250623 02:00:01.211591 17536 consensus_queue.cc:1035] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250623 02:00:01.246899 13926 consensus_peers.cc:487] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.292739 13926 consensus_peers.cc:487] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.299561 13926 consensus_peers.cc:487] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250623 02:00:01.307965 13926 consensus_peers.cc:487] T 7895c36aec9944b7981b0a0ccacc6556 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 02:00:01.394692 13955 raft_consensus.cc:1273] T 32faa630311f4d8e99b289422fe771f5 P c243715fe12445019c33bbeafbaea635 [term 12 FOLLOWER]: Refusing update from remote peer ee041a8325864e788a383b8356afc652: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20250623 02:00:01.400058 17527 consensus_queue.cc:1035] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250623 02:00:01.411247 12636 thread.cc:535] Waited for 2000ms trying to join with diag-logger (tid 13834)
W20250623 02:00:01.472298 13852 consensus_peers.cc:487] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.473109 13852 consensus_peers.cc:487] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.476454 13852 consensus_peers.cc:487] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.509209 13852 consensus_peers.cc:487] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.509209 13926 consensus_peers.cc:487] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.534549 13926 consensus_peers.cc:487] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.556483 13852 consensus_peers.cc:487] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:01.569455 13852 consensus_peers.cc:487] T 5d1d49d7480d4088be0d5ed6baae9489 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:02.353231 13909 debug-util.cc:398] Leaking SignalData structure 0x7b080022c2c0 after lost signal to thread 12639
W20250623 02:00:02.354453 13909 debug-util.cc:398] Leaking SignalData structure 0x7b080033b600 after lost signal to thread 13745
W20250623 02:00:02.355865 13909 debug-util.cc:398] Leaking SignalData structure 0x7b080035eb60 after lost signal to thread 13912
W20250623 02:00:02.357029 13909 debug-util.cc:398] Leaking SignalData structure 0x7b080032e0c0 after lost signal to thread 13986
W20250623 02:00:02.358296 13909 debug-util.cc:398] Leaking SignalData structure 0x7b0800311c20 after lost signal to thread 14540
W20250623 02:00:02.411725 12636 thread.cc:535] Waited for 3000ms trying to join with diag-logger (tid 13834)
W20250623 02:00:02.866583 13926 consensus_peers.cc:487] T 3828994fa152441a95c5034d002516db P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:02.943418 13852 consensus_peers.cc:487] T ba9a759dbf744a06bcb77355efdf4085 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:02.991686 13926 consensus_peers.cc:487] T 5bd540ef429946c987f593ab36327b5a P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.039822 14479 consensus_peers.cc:487] T 1b3574beac134b2eabbd65a4f54b3ab3 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.108008 13852 consensus_peers.cc:487] T c708cd61006343f09750b88b2621eec6 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.138609 13852 consensus_peers.cc:487] T 0c8353de982c4260a5ecf1445df4ba21 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.193712 13852 consensus_peers.cc:487] T 2d3844327a1b452da89d70d856d0a80f P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.246428 13926 consensus_peers.cc:487] T 3df3a5b0b6b347b89175eef714fe6a07 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.267858 13926 consensus_peers.cc:487] T f023b481774a4d06bf5c5c62a3df3012 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.269114 13852 consensus_peers.cc:487] T adb1e25eef1743848cf218e294bc9226 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.272568 13852 consensus_peers.cc:487] T b9e5783d0e1d49b899145b9d9e01f6f9 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.287424 13926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111) [suppressed 508 similar messages]
W20250623 02:00:03.296372 14479 consensus_peers.cc:487] T 92909e3375c549b5baa0595e1750fa1f P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.304859 14479 consensus_peers.cc:487] T d2a94c049f124a68ad6678cff7f838ac P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.322542 13926 consensus_peers.cc:487] T bdaf5d40df984f96b90a4d8ace220223 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.349439 14479 consensus_peers.cc:487] T 829478b3e1dc4a95a122fcd8be3d2c77 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.356074 13852 consensus_peers.cc:487] T f44146f651124d8fba4cc1825f480603 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.358255 13926 consensus_peers.cc:487] T 02e07b6deb3546fcad55d41d2ecd8a5a P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.387096 14479 consensus_peers.cc:487] T 819094ffb4314769ab2a06512232fc61 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.404918 14479 consensus_peers.cc:487] T 0f12346da5a14097bbfa150cb8d1d907 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.412232 12636 thread.cc:535] Waited for 4000ms trying to join with diag-logger (tid 13834)
W20250623 02:00:03.415828 13926 consensus_peers.cc:487] T 48dd5f33321544e58bc600dbff358693 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.448396 13852 consensus_peers.cc:487] T c078b30fc50b475193ae485fbe10e04c P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.461211 14479 consensus_peers.cc:487] T 2540092c63cb4d0cbe07ae4018f99f46 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.467097 14479 consensus_peers.cc:487] T e2928af9e5df4f9eacf59f499ab96dc1 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.468050 13926 consensus_peers.cc:487] T dfe9e13c26c84532b389e343edf7cef4 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.486272 13926 consensus_peers.cc:487] T 939deead84d54fe0b858c13a3dfaf686 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.490700 13852 consensus_peers.cc:487] T 7560a94e265b433dbc427ef13c120669 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.493361 13852 consensus_peers.cc:487] T ff8dea72f9f9447abaa5fa803f5e84e8 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.512987 14479 consensus_peers.cc:487] T 318e5309795f47f3868e0787ddfdd146 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.492725 13926 consensus_peers.cc:487] T 05203153a0ed43df91fde4fe090579a7 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.522332 13852 consensus_peers.cc:487] T bb4438d5ce9e41f3a72f00de14912cff P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.525528 13926 consensus_peers.cc:487] T 839cb616e68044308dfa7e08eadc1903 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.534353 13926 consensus_peers.cc:487] T 6f13ab3fda6f4c6b9b68d33e7a89c988 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.546422 13852 consensus_peers.cc:487] T 86045b40aeea4a02993bcd65deecd670 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.554864 14479 consensus_peers.cc:487] T 272c03e6252d489cbf2fdceb36b05dc1 P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.594831 13926 consensus_peers.cc:487] T e20d8c11ac244feeaecb254a76559f76 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.613734 13852 consensus_peers.cc:487] T 1aa785c3afe64dbcac3b658b49195b8c P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.614516 13852 consensus_peers.cc:487] T 9cb699d5420143efa03140964d5aa739 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.650010 13852 consensus_peers.cc:487] T a71ef18fd45046578f969e8e8905e897 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.683471 13926 consensus_peers.cc:487] T ade11e463c53492f8c17cba4e0500ec4 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.714208 13852 consensus_peers.cc:487] T 28d0bf37b0354956a6d195c5259a3d19 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.747292 13852 consensus_peers.cc:487] T 32faa630311f4d8e99b289422fe771f5 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.750206 13852 consensus_peers.cc:487] T 7c4f89b1590c439a9b723110eaad17a8 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.757584 13926 consensus_peers.cc:487] T afad1ff768904b11936d338d33df7998 P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.763648 14479 consensus_peers.cc:487] T e92cd39500a645e79b368d26acb587ed P 7d22741cfb9b408684cb82c58241b2f5 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.778672 13926 consensus_peers.cc:487] T 5d4cc11489de46dcb7d5f77daefd49af P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.798770 13852 consensus_peers.cc:487] T cd5ada8676314a73af7cef4c5cd5553e P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.823345 13926 consensus_peers.cc:487] T 933266ad75974c449e00b527ab278f8f P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.859856 13852 consensus_peers.cc:487] T 632bb79cb33a491fa5765cd80357f1a4 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250623 02:00:03.875106 13956 consensus_queue.cc:237] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ee041a8325864e788a383b8356afc652" member_type: VOTER last_known_addr { host: "127.12.87.2" port: 43055 } } peers { permanent_uuid: "c243715fe12445019c33bbeafbaea635" member_type: VOTER last_known_addr { host: "127.12.87.3" port: 33369 } } peers { permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4" port: 39393 } attrs { promote: false } }
W20250623 02:00:03.884260 13926 consensus_peers.cc:487] T 50c92eb44c6c480d832145fd7c9c8d3f P c243715fe12445019c33bbeafbaea635 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250623 02:00:03.888247 13852 consensus_peers.cc:487] T 2f02354fc186482aa0fc28cab32390c5 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 02:00:03.891422 13852 consensus_peers.cc:487] T e773c5700b0945d1981f481534952e85 P ee041a8325864e788a383b8356afc652 -> Peer 85a58a834b6b4b0dbe197b3dad87e382 (127.12.87.1:34781): Couldn't send request to peer 85a58a834b6b4b0dbe197b3dad87e382. Status: Network error: Client connection negotiation failed: client connection to 127.12.87.1:34781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250623 02:00:03.899607 14509 raft_consensus.cc:1273] T 15e1fc4335c84e18ba2e5c6ea55fed7a P 7d22741cfb9b408684cb82c58241b2f5 [term 10 FOLLOWER]: Refusing update from remote peer c243715fe12445019c33bbeafbaea635: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20250623 02:00:03.911054 17595 consensus_queue.cc:1035] T 15e1fc4335c84e18ba2e5c6ea55fed7a P c243715fe12445019c33bbeafbaea635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d22741cfb9b408684cb82c58241b2f5" member_type: VOTER last_known_addr { host: "127.12.87.4"